You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/11/16 02:54:48 UTC
[GitHub] [pulsar] heesung-sn opened a new pull request, #18489: [improve][broker] PIP-192 Added ServiceUnitStateChannelImpl
heesung-sn opened a new pull request, #18489:
URL: https://github.com/apache/pulsar/pull/18489
<!--
### Contribution Checklist
- PR title format should be *[type][component] summary*. For details, see *[Guideline - Pulsar PR Naming Convention](https://docs.google.com/document/d/1d8Pw6ZbWk-_pCKdOmdvx9rnhPiyuxwq60_TrD68d7BA/edit#heading=h.trs9rsex3xom)*.
- Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.
- Each pull request should address only one issue, not mix up code from multiple issues.
- Each commit in the pull request has a meaningful commit message
- Once all items of the checklist are addressed, remove the above text and this checklist, leaving only the filled out template below.
-->
<!-- Either this PR fixes an issue, -->
Fixes #xyz
<!-- or this PR is one task of an issue -->
Master Issue: https://github.com/apache/pulsar/issues/16691
### Motivation
<!-- Explain here the context, and why you're making that change. What is the problem you're trying to solve. -->
We will start raising PRs to implement PIP-192, https://github.com/apache/pulsar/issues/16691
### Modifications
This PR Added ServiceUnitStateChannelImpl and its unit test.
For the pip-192 project, this ServiceUnitStateChannelImpl
- has a tableview and a producer on the system topic, persistent://pulsar/system/service-unit-state-channel
- has a leader coordination service to elect the owner broker of the channel.
- define message handlers according to the service unit state changes.
- define ownership cleanup logic in case a broker is unregistered (dead).
<!-- Describe the modifications you've done. -->
### Verifying this change
- [x] Make sure that the change passes the CI checks.
This change added tests and can be verified as follows:
- *Added unit tests for the new classes.
### Does this pull request potentially affect one of the following parts:
*If the box was checked, please highlight the changes*
- [ ] Dependencies (add or upgrade a dependency)
- [ ] The public API
- [ ] The schema
- [ ] The default values of configurations
- [ ] The threading model
- [ ] The binary protocol
- [ ] The REST endpoints
- [ ] The admin CLI options
- [ ] Anything that affects deployment
### Documentation
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
- [ ] `doc` <!-- Your PR contains doc changes. Please attach the local preview screenshots (run `sh start.sh` at `pulsar/site2/website`) to your PR description, or else your PR might not get merged. -->
- [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
- [x] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->
We will have separate PRs to update the Doc later.
### Matching PR in forked repository
PR in forked repository: https://github.com/heesung-sn/pulsar/pull/14
<!--
After opening this PR, the build in apache/pulsar will fail and instructions will
be provided for opening a PR in the PR author's forked repository.
apache/pulsar pull requests should be first tested in your own fork since the
apache/pulsar CI based on GitHub Actions has constrained resources and quota.
GitHub Actions provides separate quota for pull requests that are executed in
a forked repository.
The tests will be run in the forked repository until all PR review comments have
been handled, the tests pass and the PR is approved by a reviewer.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui merged pull request #18489: [improve][broker] PIP-192 Added ServiceUnitStateChannelImpl
Posted by GitBox <gi...@apache.org>.
codelipenghui merged PR #18489:
URL: https://github.com/apache/pulsar/pull/18489
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] heesung-sn commented on a diff in pull request #18489: [improve][broker] PIP-192 Added ServiceUnitStateChannelImpl
Posted by GitBox <gi...@apache.org>.
heesung-sn commented on code in PR #18489:
URL: https://github.com/apache/pulsar/pull/18489#discussion_r1035543487
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,778 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Free;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Closed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Constructed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.LeaderElectionServiceStarted;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Started;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private final Schema<ServiceUnitStateData> schema;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private final String lookupServiceAddress;
+ // TODO: define BrokerRegistry
+ private final Semaphore outstandingCleanupTombstoneMessages;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+ private final LeaderElectionService leaderElectionService;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private ScheduledFuture<?> cleanupTasks;
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+ private volatile ChannelState channelState;
+
+ enum ChannelState {
+ Closed(0),
+ Constructed(1),
+ LeaderElectionServiceStarted(2),
+ Started(3);
+
+ ChannelState(int id) {
+ this.id = id;
+ }
+ int id;
+ }
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ this.lookupServiceAddress = pulsar.getLookupServiceAddress();
+ this.schema = Schema.JSON(ServiceUnitStateData.class);
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder().build();
+ this.cleanupJobs = ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder().build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} is the leader now.", lookupServiceAddress);
+ // TODO: schedule monitorOwnerships by brokerRegistry
+ } else {
+ log.debug("This broker:{} is a follower now.", lookupServiceAddress);
+ // TODO: cancel scheduled monitorOwnerships if any
+ }
+ });
+ this.channelState = Constructed;
+ }
+
+ public synchronized void start() throws PulsarServerException {
+ validateChannelState(LeaderElectionServiceStarted, false);
+ try {
+ leaderElectionService.start();
+ this.channelState = LeaderElectionServiceStarted;
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(schema)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(schema)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ channelState = Started;
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public synchronized void close() throws PulsarServerException {
+ channelState = Closed;
+ try {
+ leaderElectionService.close();
+ log.debug("Successfully closed the channel leader election service.");
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ private void validateChannelState(ChannelState targetState, boolean checkLowerIds) {
+ int order = checkLowerIds ? -1 : 1;
+ if (Integer.compare(channelState.id, targetState.id) * order > 0) {
+ throw new IllegalStateException("Invalid channel state:" + channelState.name());
+ }
+ }
+
+ public synchronized void scheduleCompaction() throws PulsarServerException {
+ try {
+ Long threshold = pulsar.getAdminClient().topicPolicies()
+ .getCompactionThreshold(TOPIC);
+ if (threshold == null || threshold == 0) {
+ pulsar.getAdminClient().topicPolicies()
+ .setCompactionThreshold(TOPIC, COMPACTION_THRESHOLD);
+ log.info("Scheduled compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ } else {
+ log.info("Already set compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ }
+ } catch (PulsarAdminException e) {
+ throw new PulsarServerException("Failed to schedule compaction.", e);
+ }
+ }
+
+ public String getChannelOwner() {
+ validateChannelState(LeaderElectionServiceStarted, true);
+ CompletableFuture<Optional<LeaderBroker>> future = leaderElectionService.readCurrentLeader();
+ if (!future.isDone()) {
+ return null;
+ }
+ Optional<LeaderBroker> leader = future.join();
+ if (leader.isEmpty()) {
+ return null;
+ }
+ //expecting http://broker-xyz:port
+ // TODO: discard this protocol prefix removal by a util func that returns lookupServiceAddress(serviceUrl)
+ String broker = leader.get().getServiceUrl();
+ broker = broker.substring(broker.lastIndexOf('/') + 1);
+ return broker;
+ }
+
+ public boolean isChannelOwner() {
+ return isTargetBroker(getChannelOwner());
+ }
+
+ public CompletableFuture<String> getOwnerAsync(String serviceUnit) {
+ validateChannelState(Started, true);
+ ServiceUnitStateData data = tableview.get(serviceUnit);
+ if (data == null) {
+ return null;
+ }
+ switch (data.state()) {
+ case Owned, Splitting -> {
+ return CompletableFuture.completedFuture(data.broker());
+ }
+ case Assigned, Released -> {
+ return deferGetOwnerRequest(serviceUnit);
+ }
+ default -> {
+ return null;
+ }
+ }
+ }
+
+ public CompletableFuture<String> publishAssignEventAsync(String serviceUnit, String broker) {
+ CompletableFuture<String> getOwnerRequest = deferGetOwnerRequest(serviceUnit);
+ pubAsync(serviceUnit, new ServiceUnitStateData(Assigned, broker))
+ .whenComplete((__, ex) -> {
+ if (ex != null) {
+ getOwnerRequests.remove(serviceUnit);
+ if (!getOwnerRequest.isCompletedExceptionally()) {
+ getOwnerRequest.completeExceptionally(ex);
+ }
+ }
+ });
+
+ return getOwnerRequest;
+ }
+
+ public CompletableFuture<Void> publishUnloadEventAsync(Unload unload) {
+ String serviceUnit = unload.serviceUnit();
+ if (isTransferCommand(unload)) {
+ ServiceUnitStateData next = new ServiceUnitStateData(Assigned,
+ unload.destBroker().get(), unload.sourceBroker());
+ return pubAsync(serviceUnit, next).thenAccept(__ -> {
+ });
+ }
+ return tombstoneAsync(serviceUnit).thenAccept(__ -> {
+ });
+ }
+
+ public CompletableFuture<Void> publishSplitEventAsync(Split split) {
+ String serviceUnit = split.serviceUnit();
+ ServiceUnitStateData next = new ServiceUnitStateData(Splitting, split.sourceBroker());
+ return pubAsync(serviceUnit, next).thenAccept(__ -> {
+ });
+ }
+
+ private void handle(String serviceUnit, ServiceUnitStateData data) {
+ if (log.isDebugEnabled()) {
+ log.info("{} received a handle request for serviceUnit:{}, data:{}",
+ lookupServiceAddress, serviceUnit, data);
+ }
+
+ ServiceUnitState state = data == null ? Free : data.state();
+
+ // TODO : Add state validation in tableview by the compaction strategy
+ switch (state) {
+ case Owned -> handleOwnEvent(serviceUnit, data);
+ case Assigned -> handleAssignEvent(serviceUnit, data);
+ case Released -> handleReleaseEvent(serviceUnit, data);
+ case Splitting -> handleSplitEvent(serviceUnit, data);
+ case Free -> handleFreeEvent(serviceUnit);
+ default -> throw new IllegalStateException("Failed to handle channel data:" + data);
+ }
+ }
+
+ private static boolean isTransferCommand(ServiceUnitStateData data) {
+ if (data == null) {
+ return false;
+ }
+ return StringUtils.isNotEmpty(data.sourceBroker());
+ }
+
+ private static boolean isTransferCommand(Unload data) {
+ return data.destBroker().isPresent();
+ }
+
+ private static String getLogEventTag(ServiceUnitStateData data) {
+ return data == null ? "Free" :
+ isTransferCommand(data) ? "Transfer:" + data.state() : data.state().toString();
+ }
+
+ private void log(Throwable e, String serviceUnit, ServiceUnitStateData data, ServiceUnitStateData next) {
+ if (e == null) {
+ if (log.isDebugEnabled() || isTransferCommand(data)) {
+ log.info("{} handled {} event for serviceUnit:{}, cur:{}, next:{}",
+ lookupServiceAddress, getLogEventTag(data), serviceUnit,
+ data == null ? "" : data,
+ next == null ? "" : next);
+ }
+ } else {
+ log.error("{} failed to handle {} event for serviceUnit:{}, cur:{}, next:{}",
+ lookupServiceAddress, getLogEventTag(data), serviceUnit,
+ data == null ? "" : data,
+ next == null ? "" : next,
+ e);
+ }
+ }
+
+ private void handleOwnEvent(String serviceUnit, ServiceUnitStateData data) {
+ var getOwnerRequest = getOwnerRequests.remove(serviceUnit);
+ if (getOwnerRequest != null) {
+ getOwnerRequest.complete(data.broker());
+ }
+ if (isTargetBroker(data.broker())) {
+ log(null, serviceUnit, data, null);
+ }
+
+ }
+
+ private void handleAssignEvent(String serviceUnit, ServiceUnitStateData data) {
+ deferGetOwnerRequest(serviceUnit);
+ if (isTargetBroker(data.broker())) {
+ ServiceUnitStateData next = new ServiceUnitStateData(
+ isTransferCommand(data) ? Released : Owned, data.broker(), data.sourceBroker());
+ pubAsync(serviceUnit, next)
+ .whenComplete((__, e) -> log(e, serviceUnit, data, next));
+ }
+ }
+
+ private void handleReleaseEvent(String serviceUnit, ServiceUnitStateData data) {
+
+ if (isTargetBroker(data.sourceBroker())) {
+ ServiceUnitStateData next = new ServiceUnitStateData(Owned, data.broker(), data.sourceBroker());
+ // TODO: when close, pass message to clients to connect to the new broker
+ closeServiceUnit(serviceUnit)
+ .thenCompose(__ -> pubAsync(serviceUnit, next))
+ .whenComplete((__, e) -> log(e, serviceUnit, data, next));
+ }
+ }
+
+ private void handleSplitEvent(String serviceUnit, ServiceUnitStateData data) {
+ if (isTargetBroker(data.broker())) {
+ splitServiceUnit(serviceUnit)
+ .thenCompose(__ -> tombstoneAsync(serviceUnit))
+ .whenComplete((__, e) -> log(e, serviceUnit, data, null));
+ }
+ }
+
+ private void handleFreeEvent(String serviceUnit) {
+ closeServiceUnit(serviceUnit)
+ .thenAccept(__ -> {
+ var request = getOwnerRequests.remove(serviceUnit);
+ if (request != null) {
+ request.completeExceptionally(new IllegalStateException("The ownership has been unloaded. "
+ + "No owner is found for serviceUnit: " + serviceUnit));
+ }
+ })
+ .whenComplete((__, e) -> log(e, serviceUnit, null, null));
+ }
+
+ private CompletableFuture<MessageId> pubAsync(String serviceUnit, ServiceUnitStateData data) {
+ validateChannelState(Started, true);
+ CompletableFuture<MessageId> future = new CompletableFuture<>();
+ producer.newMessage()
+ .key(serviceUnit)
+ .value(data)
+ .sendAsync()
+ .whenComplete((messageId, e) -> {
+ if (e != null) {
+ log.error("Failed to publish the message: serviceUnit:{}, data:{}",
+ serviceUnit, data, e);
+ future.completeExceptionally(e);
+ } else {
+ future.complete(messageId);
+ }
+ });
+ return future;
+ }
+
+ private CompletableFuture<MessageId> tombstoneAsync(String serviceUnit) {
+ return pubAsync(serviceUnit, null);
+ }
+
+ private boolean isTargetBroker(String broker) {
+ if (broker == null) {
+ return false;
+ }
+ return broker.equals(lookupServiceAddress);
+ }
+
+ private NamespaceBundle getNamespaceBundle(String bundle) {
+ final String namespaceName = LoadManagerShared.getNamespaceNameFromBundleName(bundle);
+ final String bundleRange = LoadManagerShared.getBundleRangeFromBundleName(bundle);
+ return pulsar.getNamespaceService().getNamespaceBundleFactory().getBundle(namespaceName, bundleRange);
+ }
+
+ private CompletableFuture<String> deferGetOwnerRequest(String serviceUnit) {
+ return getOwnerRequests
+ .computeIfAbsent(serviceUnit, k -> {
+ CompletableFuture future = new CompletableFuture<>();
+ future.orTimeout(inFlightStateWaitingTimeInMillis, TimeUnit.MILLISECONDS)
+ .whenComplete((v, e) -> {
+ if (e != null) {
+ getOwnerRequests.remove(serviceUnit);
+ log.warn("Failed to getOwner for serviceUnit:{}",
+ serviceUnit, e);
+ future.completeExceptionally((Throwable) e);
Review Comment:
I see what you meant. Thank you for the clarification. Updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] heesung-sn commented on a diff in pull request #18489: [improve][broker] PIP-192 Added ServiceUnitStateChannelImpl
Posted by GitBox <gi...@apache.org>.
heesung-sn commented on code in PR #18489:
URL: https://github.com/apache/pulsar/pull/18489#discussion_r1027241928
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,779 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ private static final Schema<ServiceUnitStateData> SCHEMA = Schema.JSON(ServiceUnitStateData.class);
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private String lookupServiceAddress;
+ private LeaderElectionService leaderElectionService;
+ // TODO: define BrokerRegistry
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private Semaphore outstandingPubMessages;
+
+ private Semaphore outstandingCleanupTombstoneMessages;
+ private ScheduledFuture<?> cleanupTasks;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+
+ private ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ ServiceConfiguration conf = pulsar.getConfiguration();
+ this.lookupServiceAddress = pulsar.getAdvertisedAddress() + ":"
+ + (conf.getWebServicePort().isPresent() ? conf.getWebServicePort().get()
+ : conf.getWebServicePortTls().get());
+ this.outstandingPubMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder()
+ .build();
+ this.cleanupJobs =
+ ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder()
+ .build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ }
+
+ public void start() throws PulsarServerException {
+
+ try {
+ if (leaderElectionService != null) {
+ leaderElectionService.close();
+ log.debug("Closed the channel leader election service.");
+ }
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} was elected as the leader."
+ + "Current channel leader is {}",
+ lookupServiceAddress,
+ leaderElectionService.getCurrentLeader());
+ } else {
+ if (leaderElectionService != null) {
+ log.debug("This broker:{} is a follower. "
+ + "Current channel leader is {}",
+ lookupServiceAddress,
+ leaderElectionService.getCurrentLeader());
+ }
+ }
+ });
+ leaderElectionService.start();
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(SCHEMA)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(SCHEMA)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ // TODO: schedule cleanupTasks by brokerRegistry
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+
+ public void close() throws PulsarServerException {
+ try {
+ if (leaderElectionService != null) {
+ leaderElectionService.close();
+ leaderElectionService = null;
+ log.debug("Successfully closed the channel leader election service.");
+ }
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public void scheduleCompaction() throws PulsarServerException {
+ try {
+ Long threshold = pulsar.getAdminClient().topicPolicies()
+ .getCompactionThreshold(TOPIC);
+ if (threshold == null || threshold == 0) {
+ pulsar.getAdminClient().topicPolicies()
+ .setCompactionThreshold(TOPIC, COMPACTION_THRESHOLD);
+ log.info("Scheduled compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ } else {
+ log.info("Already set compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ }
+ } catch (PulsarAdminException e) {
+ throw new PulsarServerException("Failed to schedule compaction.", e);
+ }
+ }
+
+ public String getChannelOwner() {
+ CompletableFuture<Optional<LeaderBroker>> future = leaderElectionService.readCurrentLeader();
Review Comment:
About the concurrent access to the member variables, still 'tableview' and 'producer' are not final.
Could you explain more why we need the synchronized access to the 'leaderElectionService' instance?
As long as it's non-null, shouldn't it be ok whether its from the old or new instance of 'leaderElectionService'?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] heesung-sn commented on a diff in pull request #18489: [improve][broker] PIP-192 Added ServiceUnitStateChannelImpl
Posted by GitBox <gi...@apache.org>.
heesung-sn commented on code in PR #18489:
URL: https://github.com/apache/pulsar/pull/18489#discussion_r1034326793
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,778 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Free;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Closed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Constructed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.LeaderElectionServiceStarted;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Started;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private final Schema<ServiceUnitStateData> schema;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private final String lookupServiceAddress;
+ // TODO: define BrokerRegistry
+ private final Semaphore outstandingCleanupTombstoneMessages;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+ private final LeaderElectionService leaderElectionService;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private ScheduledFuture<?> cleanupTasks;
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+ private volatile ChannelState channelState;
+
+ enum ChannelState {
+ Closed(0),
+ Constructed(1),
+ LeaderElectionServiceStarted(2),
+ Started(3);
+
+ ChannelState(int id) {
+ this.id = id;
+ }
+ int id;
+ }
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ this.lookupServiceAddress = pulsar.getLookupServiceAddress();
+ this.schema = Schema.JSON(ServiceUnitStateData.class);
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder().build();
+ this.cleanupJobs = ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder().build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} is the leader now.", lookupServiceAddress);
+ // TODO: schedule monitorOwnerships by brokerRegistry
+ } else {
+ log.debug("This broker:{} is a follower now.", lookupServiceAddress);
+ // TODO: cancel scheduled monitorOwnerships if any
+ }
+ });
+ this.channelState = Constructed;
+ }
+
+ public synchronized void start() throws PulsarServerException {
+ validateChannelState(LeaderElectionServiceStarted, false);
+ try {
+ leaderElectionService.start();
+ this.channelState = LeaderElectionServiceStarted;
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(schema)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(schema)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ channelState = Started;
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public synchronized void close() throws PulsarServerException {
+ channelState = Closed;
+ try {
+ leaderElectionService.close();
+ log.debug("Successfully closed the channel leader election service.");
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ private void validateChannelState(ChannelState targetState, boolean checkLowerIds) {
+ int order = checkLowerIds ? -1 : 1;
+ if (Integer.compare(channelState.id, targetState.id) * order > 0) {
+ throw new IllegalStateException("Invalid channel state:" + channelState.name());
+ }
+ }
+
+ public synchronized void scheduleCompaction() throws PulsarServerException {
+ try {
+ Long threshold = pulsar.getAdminClient().topicPolicies()
+ .getCompactionThreshold(TOPIC);
+ if (threshold == null || threshold == 0) {
+ pulsar.getAdminClient().topicPolicies()
+ .setCompactionThreshold(TOPIC, COMPACTION_THRESHOLD);
+ log.info("Scheduled compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ } else {
+ log.info("Already set compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ }
+ } catch (PulsarAdminException e) {
+ throw new PulsarServerException("Failed to schedule compaction.", e);
+ }
+ }
+
+ public String getChannelOwner() {
+ validateChannelState(LeaderElectionServiceStarted, true);
+ CompletableFuture<Optional<LeaderBroker>> future = leaderElectionService.readCurrentLeader();
+ if (!future.isDone()) {
+ return null;
+ }
+ Optional<LeaderBroker> leader = future.join();
+ if (leader.isEmpty()) {
+ return null;
+ }
+ //expecting http://broker-xyz:port
+ // TODO: discard this protocol prefix removal by a util func that returns lookupServiceAddress(serviceUrl)
+ String broker = leader.get().getServiceUrl();
+ broker = broker.substring(broker.lastIndexOf('/') + 1);
+ return broker;
+ }
+
+ public boolean isChannelOwner() {
+ return isTargetBroker(getChannelOwner());
+ }
+
+ public CompletableFuture<String> getOwnerAsync(String serviceUnit) {
+ validateChannelState(Started, true);
+ ServiceUnitStateData data = tableview.get(serviceUnit);
+ if (data == null) {
+ return null;
+ }
+ switch (data.state()) {
+ case Owned, Splitting -> {
+ return CompletableFuture.completedFuture(data.broker());
+ }
+ case Assigned, Released -> {
+ return deferGetOwnerRequest(serviceUnit);
+ }
+ default -> {
+ return null;
+ }
+ }
+ }
+
+ public CompletableFuture<String> publishAssignEventAsync(String serviceUnit, String broker) {
+ CompletableFuture<String> getOwnerRequest = deferGetOwnerRequest(serviceUnit);
+ pubAsync(serviceUnit, new ServiceUnitStateData(Assigned, broker))
+ .whenComplete((__, ex) -> {
+ if (ex != null) {
+ getOwnerRequests.remove(serviceUnit);
+ if (!getOwnerRequest.isCompletedExceptionally()) {
+ getOwnerRequest.completeExceptionally(ex);
+ }
+ }
+ });
+
+ return getOwnerRequest;
+ }
+
+ public CompletableFuture<Void> publishUnloadEventAsync(Unload unload) {
+ String serviceUnit = unload.serviceUnit();
+ if (isTransferCommand(unload)) {
+ ServiceUnitStateData next = new ServiceUnitStateData(Assigned,
+ unload.destBroker().get(), unload.sourceBroker());
+ return pubAsync(serviceUnit, next).thenAccept(__ -> {
+ });
+ }
+ return tombstoneAsync(serviceUnit).thenAccept(__ -> {
+ });
+ }
+
+ public CompletableFuture<Void> publishSplitEventAsync(Split split) {
+ String serviceUnit = split.serviceUnit();
+ ServiceUnitStateData next = new ServiceUnitStateData(Splitting, split.sourceBroker());
+ return pubAsync(serviceUnit, next).thenAccept(__ -> {
+ });
+ }
+
+ private void handle(String serviceUnit, ServiceUnitStateData data) {
+ if (log.isDebugEnabled()) {
+ log.info("{} received a handle request for serviceUnit:{}, data:{}",
+ lookupServiceAddress, serviceUnit, data);
+ }
+
+ ServiceUnitState state = data == null ? Free : data.state();
+
+ // TODO : Add state validation in tableview by the compaction strategy
+ switch (state) {
+ case Owned -> handleOwnEvent(serviceUnit, data);
+ case Assigned -> handleAssignEvent(serviceUnit, data);
+ case Released -> handleReleaseEvent(serviceUnit, data);
+ case Splitting -> handleSplitEvent(serviceUnit, data);
+ case Free -> handleFreeEvent(serviceUnit);
+ default -> throw new IllegalStateException("Failed to handle channel data:" + data);
+ }
+ }
+
+ private static boolean isTransferCommand(ServiceUnitStateData data) {
+ if (data == null) {
+ return false;
+ }
+ return StringUtils.isNotEmpty(data.sourceBroker());
+ }
+
+ private static boolean isTransferCommand(Unload data) {
+ return data.destBroker().isPresent();
+ }
+
+ private static String getLogEventTag(ServiceUnitStateData data) {
+ return data == null ? "Free" :
+ isTransferCommand(data) ? "Transfer:" + data.state() : data.state().toString();
+ }
+
+ private void log(Throwable e, String serviceUnit, ServiceUnitStateData data, ServiceUnitStateData next) {
+ if (e == null) {
+ if (log.isDebugEnabled() || isTransferCommand(data)) {
+ log.info("{} handled {} event for serviceUnit:{}, cur:{}, next:{}",
+ lookupServiceAddress, getLogEventTag(data), serviceUnit,
+ data == null ? "" : data,
+ next == null ? "" : next);
+ }
+ } else {
+ log.error("{} failed to handle {} event for serviceUnit:{}, cur:{}, next:{}",
+ lookupServiceAddress, getLogEventTag(data), serviceUnit,
+ data == null ? "" : data,
+ next == null ? "" : next,
+ e);
+ }
+ }
+
+ private void handleOwnEvent(String serviceUnit, ServiceUnitStateData data) {
+ var getOwnerRequest = getOwnerRequests.remove(serviceUnit);
+ if (getOwnerRequest != null) {
+ getOwnerRequest.complete(data.broker());
+ }
+ if (isTargetBroker(data.broker())) {
+ log(null, serviceUnit, data, null);
+ }
+
+ }
+
+ private void handleAssignEvent(String serviceUnit, ServiceUnitStateData data) {
+ deferGetOwnerRequest(serviceUnit);
+ if (isTargetBroker(data.broker())) {
+ ServiceUnitStateData next = new ServiceUnitStateData(
+ isTransferCommand(data) ? Released : Owned, data.broker(), data.sourceBroker());
+ pubAsync(serviceUnit, next)
+ .whenComplete((__, e) -> log(e, serviceUnit, data, next));
+ }
+ }
+
+ private void handleReleaseEvent(String serviceUnit, ServiceUnitStateData data) {
+
+ if (isTargetBroker(data.sourceBroker())) {
+ ServiceUnitStateData next = new ServiceUnitStateData(Owned, data.broker(), data.sourceBroker());
+ // TODO: when close, pass message to clients to connect to the new broker
+ closeServiceUnit(serviceUnit)
+ .thenCompose(__ -> pubAsync(serviceUnit, next))
+ .whenComplete((__, e) -> log(e, serviceUnit, data, next));
+ }
+ }
+
+ private void handleSplitEvent(String serviceUnit, ServiceUnitStateData data) {
+ if (isTargetBroker(data.broker())) {
+ splitServiceUnit(serviceUnit)
+ .thenCompose(__ -> tombstoneAsync(serviceUnit))
+ .whenComplete((__, e) -> log(e, serviceUnit, data, null));
+ }
+ }
+
+ private void handleFreeEvent(String serviceUnit) {
+ closeServiceUnit(serviceUnit)
+ .thenAccept(__ -> {
+ var request = getOwnerRequests.remove(serviceUnit);
+ if (request != null) {
+ request.completeExceptionally(new IllegalStateException("The ownership has been unloaded. "
+ + "No owner is found for serviceUnit: " + serviceUnit));
+ }
+ })
+ .whenComplete((__, e) -> log(e, serviceUnit, null, null));
+ }
+
+ private CompletableFuture<MessageId> pubAsync(String serviceUnit, ServiceUnitStateData data) {
+ validateChannelState(Started, true);
+ CompletableFuture<MessageId> future = new CompletableFuture<>();
+ producer.newMessage()
+ .key(serviceUnit)
+ .value(data)
+ .sendAsync()
+ .whenComplete((messageId, e) -> {
+ if (e != null) {
+ log.error("Failed to publish the message: serviceUnit:{}, data:{}",
+ serviceUnit, data, e);
+ future.completeExceptionally(e);
+ } else {
+ future.complete(messageId);
+ }
+ });
+ return future;
+ }
+
+ private CompletableFuture<MessageId> tombstoneAsync(String serviceUnit) {
+ return pubAsync(serviceUnit, null);
+ }
+
+ private boolean isTargetBroker(String broker) {
+ if (broker == null) {
+ return false;
+ }
+ return broker.equals(lookupServiceAddress);
+ }
+
+ private NamespaceBundle getNamespaceBundle(String bundle) {
+ final String namespaceName = LoadManagerShared.getNamespaceNameFromBundleName(bundle);
+ final String bundleRange = LoadManagerShared.getBundleRangeFromBundleName(bundle);
+ return pulsar.getNamespaceService().getNamespaceBundleFactory().getBundle(namespaceName, bundleRange);
+ }
+
+ private CompletableFuture<String> deferGetOwnerRequest(String serviceUnit) {
+ return getOwnerRequests
+ .computeIfAbsent(serviceUnit, k -> {
+ CompletableFuture future = new CompletableFuture<>();
+ future.orTimeout(inFlightStateWaitingTimeInMillis, TimeUnit.MILLISECONDS)
+ .whenComplete((v, e) -> {
+ if (e != null) {
+ getOwnerRequests.remove(serviceUnit);
+ log.warn("Failed to getOwner for serviceUnit:{}",
+ serviceUnit, e);
+ future.completeExceptionally((Throwable) e);
+ }
+ }
+ );
+ return future;
+ });
+ }
+
+ private CompletableFuture<Integer> closeServiceUnit(String serviceUnit) {
+ long startTime = System.nanoTime();
+ MutableInt unloadedTopics = new MutableInt();
+ NamespaceBundle bundle = getNamespaceBundle(serviceUnit);
+ return pulsar.getBrokerService().unloadServiceUnit(
+ bundle,
+ false,
+ pulsar.getConfig().getNamespaceBundleUnloadingTimeoutMs(),
+ TimeUnit.MILLISECONDS)
+ .handle((numUnloadedTopics, ex) -> {
+ if (ex != null) {
+ // ignore topic-close failure to unload bundle
+ log.error("Failed to close topics under bundle:{}",
+ bundle.toString(), ex);
+ } else {
+ unloadedTopics.setValue(numUnloadedTopics);
+ }
+ // clean up topics that failed to unload from the broker ownership cache
+ pulsar.getBrokerService().cleanUnloadedTopicFromCache(bundle);
+ return numUnloadedTopics;
+ })
+ .whenComplete((ignored, ex) -> {
+ double unloadBundleTime = TimeUnit.NANOSECONDS
+ .toMillis((System.nanoTime() - startTime));
+ log.info("Unloading bundle:{} with {} topics completed in {} ms",
+ bundle, unloadedTopics, unloadBundleTime, ex);
+ });
+ }
+
+ private CompletableFuture<Void> splitServiceUnit(String serviceUnit) {
+ // TODO: after the split we need to write the child ownerships to BSC instead of ZK.
+ // Use the configured NamespaceBundleSplitAlgorithm
+ long startTime = System.nanoTime();
+ return pulsar.getNamespaceService()
+ .splitAndOwnBundle(getNamespaceBundle(serviceUnit),
+ false,
+ NamespaceBundleSplitAlgorithm.RANGE_EQUALLY_DIVIDE_ALGO,
+ null)
+ .whenComplete((__, ex) -> {
+ double splitBundleTime = TimeUnit.NANOSECONDS
+ .toMillis((System.nanoTime() - startTime));
+ if (ex == null) {
+ log.info("Successfully split {} namespace-bundle in {} ms",
+ serviceUnit, splitBundleTime);
+ } else {
+ log.error("Failed to split {} namespace-bundle in {} ms",
+ serviceUnit, splitBundleTime, ex);
+ }
+ });
+ }
+
+ public void handleMetadataSessionEvent(SessionEvent e) {
+ if (e == SessionReestablished || e == SessionLost) {
+ lastMetadataSessionEvent = e;
+ lastMetadataSessionEventTimestamp = System.currentTimeMillis();
+ log.info("Received metadata session event:{} at timestamp:{}",
+ lastMetadataSessionEvent, lastMetadataSessionEventTimestamp);
+ }
+ }
+
+ public void handleBrokerRegistrationEvent(String broker, NotificationType type) {
+ if (type == NotificationType.Created) {
+ handleBrokerCreationEvent(broker);
+ } else if (type == NotificationType.Deleted) {
+ handleBrokerDeletionEvent(broker);
+ }
+ }
+
+ private MetadataState getMetadataState() {
+ long now = System.currentTimeMillis();
+ if (lastMetadataSessionEvent == SessionReestablished) {
+ if (now - lastMetadataSessionEventTimestamp > 1000 * maxCleanupDelayTimeInSecs) {
+ return Stable;
+ }
+ return Jittery;
+ }
+ return Unstable;
+ }
+
+ private void handleBrokerCreationEvent(String broker) {
+ CompletableFuture<Void> future = cleanupJobs.remove(broker);
+ if (future != null) {
+ future.cancel(false);
+ totalCleanupCancelledCnt++;
+ log.info("Successfully cancelled the ownership cleanup for broker:{}. isChannelOwner:{}."
+ + " Active cleanup job count:{}",
+ broker, isChannelOwner(), cleanupJobs.size());
+ } else {
+ log.info("Failed to cancel the ownership cleanup for broker:{}. isChannelOwner:{}."
+ + " There was no scheduled cleanup job. Active cleanup job count:{}",
+ broker, isChannelOwner(), cleanupJobs.size());
+ }
+ }
+
+ private void handleBrokerDeletionEvent(String broker) {
+ if (!isChannelOwner()) {
+ return;
+ }
+ MetadataState state = getMetadataState();
+ log.info("Handling broker:{} ownership cleanup based on metadata connection state:{}, event:{}, event_ts:{}:",
+ broker, state, lastMetadataSessionEvent, lastMetadataSessionEventTimestamp);
+ switch (state) {
+ case Stable -> scheduleCleanup(broker, minCleanupDelayTimeInSecs);
+ case Jittery -> scheduleCleanup(broker, maxCleanupDelayTimeInSecs);
+ case Unstable -> {
+ totalCleanupIgnoredCnt++;
+ log.error("MetadataState state is unstable. "
+ + "Ignoring the ownership cleanup request for the reported broker :{}", broker);
+ }
+ }
+ }
+
+ private void scheduleCleanup(String broker, long delayInSecs) {
+ cleanupJobs.computeIfAbsent(broker, k -> {
+ Executor delayed = CompletableFuture
+ .delayedExecutor(delayInSecs, TimeUnit.SECONDS, pulsar.getLoadManagerExecutor());
+ totalCleanupScheduledCnt++;
+ return CompletableFuture
+ .runAsync(() -> doCleanup(broker), delayed);
+ });
+
+ log.info("Scheduled ownership cleanup for broker:{} with delay:{} secs. Pending clean jobs:{}.",
+ broker, delayInSecs, cleanupJobs.size());
+ }
+
+
+ private void doCleanup(String broker) {
+ long startTime = System.nanoTime();
+ log.info("Started ownership cleanup for the inactive broker:{}", broker);
+ AtomicInteger serviceUnitTombstoneCnt = new AtomicInteger();
+ AtomicInteger serviceUnitTombstoneErrorCnt = new AtomicInteger();
+ for (Map.Entry<String, ServiceUnitStateData> etr : tableview.entrySet()) {
+ ServiceUnitStateData stateData = etr.getValue();
+ String serviceUnit = etr.getKey();
+ if (StringUtils.equals(broker, stateData.broker())
+ || StringUtils.equals(broker, stateData.sourceBroker())) {
+ try {
+ outstandingCleanupTombstoneMessages.acquire();
Review Comment:
Ack.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on a diff in pull request #18489: [improve][broker] PIP-192 Added ServiceUnitStateChannelImpl
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on code in PR #18489:
URL: https://github.com/apache/pulsar/pull/18489#discussion_r1030373276
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,778 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Free;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Closed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Constructed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.LeaderElectionServiceStarted;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Started;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
Review Comment:
```suggestion
public static final String TOPIC = TopicName.get(TopicDomain.persistent.value(), NamespaceName.SYSTEM_NAMESPACE,
"loadbalancer-service-unit-state").toString();
```
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,778 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Free;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Closed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Constructed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.LeaderElectionServiceStarted;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Started;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private final Schema<ServiceUnitStateData> schema;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private final String lookupServiceAddress;
+ // TODO: define BrokerRegistry
+ private final Semaphore outstandingCleanupTombstoneMessages;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+ private final LeaderElectionService leaderElectionService;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private ScheduledFuture<?> cleanupTasks;
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+ private volatile ChannelState channelState;
+
+ enum ChannelState {
+ Closed(0),
+ Constructed(1),
+ LeaderElectionServiceStarted(2),
+ Started(3);
+
+ ChannelState(int id) {
+ this.id = id;
+ }
+ int id;
+ }
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ this.lookupServiceAddress = pulsar.getLookupServiceAddress();
+ this.schema = Schema.JSON(ServiceUnitStateData.class);
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder().build();
+ this.cleanupJobs = ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder().build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} is the leader now.", lookupServiceAddress);
+ // TODO: schedule monitorOwnerships by brokerRegistry
+ } else {
+ log.debug("This broker:{} is a follower now.", lookupServiceAddress);
+ // TODO: cancel scheduled monitorOwnerships if any
+ }
+ });
+ this.channelState = Constructed;
+ }
+
+ public synchronized void start() throws PulsarServerException {
+ validateChannelState(LeaderElectionServiceStarted, false);
+ try {
+ leaderElectionService.start();
+ this.channelState = LeaderElectionServiceStarted;
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(schema)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(schema)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ channelState = Started;
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public synchronized void close() throws PulsarServerException {
+ channelState = Closed;
+ try {
+ leaderElectionService.close();
+ log.debug("Successfully closed the channel leader election service.");
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ private void validateChannelState(ChannelState targetState, boolean checkLowerIds) {
+ int order = checkLowerIds ? -1 : 1;
+ if (Integer.compare(channelState.id, targetState.id) * order > 0) {
+ throw new IllegalStateException("Invalid channel state:" + channelState.name());
+ }
+ }
+
+ public synchronized void scheduleCompaction() throws PulsarServerException {
+ try {
+ Long threshold = pulsar.getAdminClient().topicPolicies()
+ .getCompactionThreshold(TOPIC);
+ if (threshold == null || threshold == 0) {
+ pulsar.getAdminClient().topicPolicies()
+ .setCompactionThreshold(TOPIC, COMPACTION_THRESHOLD);
+ log.info("Scheduled compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ } else {
+ log.info("Already set compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ }
+ } catch (PulsarAdminException e) {
+ throw new PulsarServerException("Failed to schedule compaction.", e);
+ }
+ }
+
+ public String getChannelOwner() {
Review Comment:
Should return `CompletableFuture<Optional<String>>`?
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,778 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Free;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Closed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Constructed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.LeaderElectionServiceStarted;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Started;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private final Schema<ServiceUnitStateData> schema;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private final String lookupServiceAddress;
+ // TODO: define BrokerRegistry
+ private final Semaphore outstandingCleanupTombstoneMessages;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+ private final LeaderElectionService leaderElectionService;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private ScheduledFuture<?> cleanupTasks;
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+ private volatile ChannelState channelState;
+
+ enum ChannelState {
+ Closed(0),
+ Constructed(1),
+ LeaderElectionServiceStarted(2),
+ Started(3);
+
+ ChannelState(int id) {
+ this.id = id;
+ }
+ int id;
+ }
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ this.lookupServiceAddress = pulsar.getLookupServiceAddress();
+ this.schema = Schema.JSON(ServiceUnitStateData.class);
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder().build();
+ this.cleanupJobs = ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder().build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} is the leader now.", lookupServiceAddress);
+ // TODO: schedule monitorOwnerships by brokerRegistry
+ } else {
+ log.debug("This broker:{} is a follower now.", lookupServiceAddress);
+ // TODO: cancel scheduled monitorOwnerships if any
+ }
+ });
+ this.channelState = Constructed;
+ }
+
+ public synchronized void start() throws PulsarServerException {
+ validateChannelState(LeaderElectionServiceStarted, false);
+ try {
+ leaderElectionService.start();
+ this.channelState = LeaderElectionServiceStarted;
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(schema)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(schema)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ channelState = Started;
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public synchronized void close() throws PulsarServerException {
+ channelState = Closed;
+ try {
+ leaderElectionService.close();
+ log.debug("Successfully closed the channel leader election service.");
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ private void validateChannelState(ChannelState targetState, boolean checkLowerIds) {
+ int order = checkLowerIds ? -1 : 1;
+ if (Integer.compare(channelState.id, targetState.id) * order > 0) {
+ throw new IllegalStateException("Invalid channel state:" + channelState.name());
+ }
+ }
+
+ public synchronized void scheduleCompaction() throws PulsarServerException {
+ try {
+ Long threshold = pulsar.getAdminClient().topicPolicies()
+ .getCompactionThreshold(TOPIC);
+ if (threshold == null || threshold == 0) {
+ pulsar.getAdminClient().topicPolicies()
+ .setCompactionThreshold(TOPIC, COMPACTION_THRESHOLD);
+ log.info("Scheduled compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ } else {
+ log.info("Already set compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ }
+ } catch (PulsarAdminException e) {
+ throw new PulsarServerException("Failed to schedule compaction.", e);
+ }
+ }
+
+ public String getChannelOwner() {
+ validateChannelState(LeaderElectionServiceStarted, true);
+ CompletableFuture<Optional<LeaderBroker>> future = leaderElectionService.readCurrentLeader();
+ if (!future.isDone()) {
+ return null;
+ }
+ Optional<LeaderBroker> leader = future.join();
+ if (leader.isEmpty()) {
+ return null;
+ }
+ //expecting http://broker-xyz:port
+ // TODO: discard this protocol prefix removal by a util func that returns lookupServiceAddress(serviceUrl)
+ String broker = leader.get().getServiceUrl();
+ broker = broker.substring(broker.lastIndexOf('/') + 1);
+ return broker;
+ }
+
+ public boolean isChannelOwner() {
+ return isTargetBroker(getChannelOwner());
+ }
+
+ public CompletableFuture<String> getOwnerAsync(String serviceUnit) {
+ validateChannelState(Started, true);
+ ServiceUnitStateData data = tableview.get(serviceUnit);
+ if (data == null) {
+ return null;
Review Comment:
We should return a completed future with null.
Please check all.
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,778 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Free;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Closed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Constructed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.LeaderElectionServiceStarted;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Started;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private final Schema<ServiceUnitStateData> schema;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private final String lookupServiceAddress;
+ // TODO: define BrokerRegistry
+ private final Semaphore outstandingCleanupTombstoneMessages;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+ private final LeaderElectionService leaderElectionService;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private ScheduledFuture<?> cleanupTasks;
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+ private volatile ChannelState channelState;
+
+ enum ChannelState {
+ Closed(0),
+ Constructed(1),
+ LeaderElectionServiceStarted(2),
+ Started(3);
+
+ ChannelState(int id) {
+ this.id = id;
+ }
+ int id;
+ }
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ this.lookupServiceAddress = pulsar.getLookupServiceAddress();
+ this.schema = Schema.JSON(ServiceUnitStateData.class);
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder().build();
+ this.cleanupJobs = ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder().build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} is the leader now.", lookupServiceAddress);
+ // TODO: schedule monitorOwnerships by brokerRegistry
+ } else {
+ log.debug("This broker:{} is a follower now.", lookupServiceAddress);
+ // TODO: cancel scheduled monitorOwnerships if any
+ }
+ });
+ this.channelState = Constructed;
+ }
+
+ public synchronized void start() throws PulsarServerException {
+ validateChannelState(LeaderElectionServiceStarted, false);
+ try {
+ leaderElectionService.start();
+ this.channelState = LeaderElectionServiceStarted;
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(schema)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(schema)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ channelState = Started;
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public synchronized void close() throws PulsarServerException {
+ channelState = Closed;
+ try {
+ leaderElectionService.close();
+ log.debug("Successfully closed the channel leader election service.");
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ private void validateChannelState(ChannelState targetState, boolean checkLowerIds) {
+ int order = checkLowerIds ? -1 : 1;
+ if (Integer.compare(channelState.id, targetState.id) * order > 0) {
+ throw new IllegalStateException("Invalid channel state:" + channelState.name());
+ }
+ }
+
+ public synchronized void scheduleCompaction() throws PulsarServerException {
+ try {
+ Long threshold = pulsar.getAdminClient().topicPolicies()
+ .getCompactionThreshold(TOPIC);
+ if (threshold == null || threshold == 0) {
+ pulsar.getAdminClient().topicPolicies()
+ .setCompactionThreshold(TOPIC, COMPACTION_THRESHOLD);
+ log.info("Scheduled compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ } else {
+ log.info("Already set compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ }
+ } catch (PulsarAdminException e) {
+ throw new PulsarServerException("Failed to schedule compaction.", e);
+ }
+ }
+
+ public String getChannelOwner() {
+ validateChannelState(LeaderElectionServiceStarted, true);
+ CompletableFuture<Optional<LeaderBroker>> future = leaderElectionService.readCurrentLeader();
+ if (!future.isDone()) {
+ return null;
+ }
+ Optional<LeaderBroker> leader = future.join();
+ if (leader.isEmpty()) {
+ return null;
+ }
+ //expecting http://broker-xyz:port
+ // TODO: discard this protocol prefix removal by a util func that returns lookupServiceAddress(serviceUrl)
+ String broker = leader.get().getServiceUrl();
+ broker = broker.substring(broker.lastIndexOf('/') + 1);
+ return broker;
+ }
+
+ public boolean isChannelOwner() {
+ return isTargetBroker(getChannelOwner());
+ }
+
+ public CompletableFuture<String> getOwnerAsync(String serviceUnit) {
+ validateChannelState(Started, true);
+ ServiceUnitStateData data = tableview.get(serviceUnit);
+ if (data == null) {
+ return null;
+ }
+ switch (data.state()) {
+ case Owned, Splitting -> {
+ return CompletableFuture.completedFuture(data.broker());
+ }
+ case Assigned, Released -> {
+ return deferGetOwnerRequest(serviceUnit);
+ }
+ default -> {
+ return null;
+ }
+ }
+ }
+
+ public CompletableFuture<String> publishAssignEventAsync(String serviceUnit, String broker) {
+ CompletableFuture<String> getOwnerRequest = deferGetOwnerRequest(serviceUnit);
+ pubAsync(serviceUnit, new ServiceUnitStateData(Assigned, broker))
+ .whenComplete((__, ex) -> {
+ if (ex != null) {
+ getOwnerRequests.remove(serviceUnit);
+ if (!getOwnerRequest.isCompletedExceptionally()) {
+ getOwnerRequest.completeExceptionally(ex);
+ }
+ }
+ });
+
+ return getOwnerRequest;
+ }
+
+ public CompletableFuture<Void> publishUnloadEventAsync(Unload unload) {
+ String serviceUnit = unload.serviceUnit();
+ if (isTransferCommand(unload)) {
+ ServiceUnitStateData next = new ServiceUnitStateData(Assigned,
+ unload.destBroker().get(), unload.sourceBroker());
+ return pubAsync(serviceUnit, next).thenAccept(__ -> {
+ });
+ }
+ return tombstoneAsync(serviceUnit).thenAccept(__ -> {
+ });
+ }
+
+ public CompletableFuture<Void> publishSplitEventAsync(Split split) {
+ String serviceUnit = split.serviceUnit();
+ ServiceUnitStateData next = new ServiceUnitStateData(Splitting, split.sourceBroker());
+ return pubAsync(serviceUnit, next).thenAccept(__ -> {
+ });
+ }
+
+ private void handle(String serviceUnit, ServiceUnitStateData data) {
+ if (log.isDebugEnabled()) {
+ log.info("{} received a handle request for serviceUnit:{}, data:{}",
+ lookupServiceAddress, serviceUnit, data);
+ }
+
+ ServiceUnitState state = data == null ? Free : data.state();
+
+ // TODO : Add state validation in tableview by the compaction strategy
+ switch (state) {
+ case Owned -> handleOwnEvent(serviceUnit, data);
+ case Assigned -> handleAssignEvent(serviceUnit, data);
+ case Released -> handleReleaseEvent(serviceUnit, data);
+ case Splitting -> handleSplitEvent(serviceUnit, data);
+ case Free -> handleFreeEvent(serviceUnit);
+ default -> throw new IllegalStateException("Failed to handle channel data:" + data);
+ }
+ }
+
+ private static boolean isTransferCommand(ServiceUnitStateData data) {
+ if (data == null) {
+ return false;
+ }
+ return StringUtils.isNotEmpty(data.sourceBroker());
+ }
+
+ private static boolean isTransferCommand(Unload data) {
+ return data.destBroker().isPresent();
+ }
+
+ private static String getLogEventTag(ServiceUnitStateData data) {
+ return data == null ? "Free" :
+ isTransferCommand(data) ? "Transfer:" + data.state() : data.state().toString();
+ }
+
+ private void log(Throwable e, String serviceUnit, ServiceUnitStateData data, ServiceUnitStateData next) {
+ if (e == null) {
+ if (log.isDebugEnabled() || isTransferCommand(data)) {
+ log.info("{} handled {} event for serviceUnit:{}, cur:{}, next:{}",
+ lookupServiceAddress, getLogEventTag(data), serviceUnit,
+ data == null ? "" : data,
+ next == null ? "" : next);
+ }
+ } else {
+ log.error("{} failed to handle {} event for serviceUnit:{}, cur:{}, next:{}",
+ lookupServiceAddress, getLogEventTag(data), serviceUnit,
+ data == null ? "" : data,
+ next == null ? "" : next,
+ e);
+ }
+ }
+
+ private void handleOwnEvent(String serviceUnit, ServiceUnitStateData data) {
+ var getOwnerRequest = getOwnerRequests.remove(serviceUnit);
+ if (getOwnerRequest != null) {
+ getOwnerRequest.complete(data.broker());
+ }
+ if (isTargetBroker(data.broker())) {
+ log(null, serviceUnit, data, null);
+ }
+
+ }
+
+ private void handleAssignEvent(String serviceUnit, ServiceUnitStateData data) {
+ deferGetOwnerRequest(serviceUnit);
+ if (isTargetBroker(data.broker())) {
+ ServiceUnitStateData next = new ServiceUnitStateData(
+ isTransferCommand(data) ? Released : Owned, data.broker(), data.sourceBroker());
+ pubAsync(serviceUnit, next)
+ .whenComplete((__, e) -> log(e, serviceUnit, data, next));
+ }
+ }
+
+ private void handleReleaseEvent(String serviceUnit, ServiceUnitStateData data) {
+
+ if (isTargetBroker(data.sourceBroker())) {
+ ServiceUnitStateData next = new ServiceUnitStateData(Owned, data.broker(), data.sourceBroker());
+ // TODO: when close, pass message to clients to connect to the new broker
+ closeServiceUnit(serviceUnit)
+ .thenCompose(__ -> pubAsync(serviceUnit, next))
+ .whenComplete((__, e) -> log(e, serviceUnit, data, next));
+ }
+ }
+
+ private void handleSplitEvent(String serviceUnit, ServiceUnitStateData data) {
+ if (isTargetBroker(data.broker())) {
+ splitServiceUnit(serviceUnit)
+ .thenCompose(__ -> tombstoneAsync(serviceUnit))
+ .whenComplete((__, e) -> log(e, serviceUnit, data, null));
+ }
+ }
+
+ private void handleFreeEvent(String serviceUnit) {
+ closeServiceUnit(serviceUnit)
+ .thenAccept(__ -> {
+ var request = getOwnerRequests.remove(serviceUnit);
+ if (request != null) {
+ request.completeExceptionally(new IllegalStateException("The ownership has been unloaded. "
+ + "No owner is found for serviceUnit: " + serviceUnit));
+ }
+ })
+ .whenComplete((__, e) -> log(e, serviceUnit, null, null));
+ }
+
+ private CompletableFuture<MessageId> pubAsync(String serviceUnit, ServiceUnitStateData data) {
+ validateChannelState(Started, true);
+ CompletableFuture<MessageId> future = new CompletableFuture<>();
+ producer.newMessage()
+ .key(serviceUnit)
+ .value(data)
+ .sendAsync()
+ .whenComplete((messageId, e) -> {
+ if (e != null) {
+ log.error("Failed to publish the message: serviceUnit:{}, data:{}",
+ serviceUnit, data, e);
+ future.completeExceptionally(e);
+ } else {
+ future.complete(messageId);
+ }
+ });
+ return future;
+ }
+
+ private CompletableFuture<MessageId> tombstoneAsync(String serviceUnit) {
+ return pubAsync(serviceUnit, null);
+ }
+
+ private boolean isTargetBroker(String broker) {
+ if (broker == null) {
+ return false;
+ }
+ return broker.equals(lookupServiceAddress);
+ }
+
+ private NamespaceBundle getNamespaceBundle(String bundle) {
+ final String namespaceName = LoadManagerShared.getNamespaceNameFromBundleName(bundle);
+ final String bundleRange = LoadManagerShared.getBundleRangeFromBundleName(bundle);
+ return pulsar.getNamespaceService().getNamespaceBundleFactory().getBundle(namespaceName, bundleRange);
+ }
+
+ private CompletableFuture<String> deferGetOwnerRequest(String serviceUnit) {
+ return getOwnerRequests
+ .computeIfAbsent(serviceUnit, k -> {
+ CompletableFuture future = new CompletableFuture<>();
Review Comment:
```suggestion
CompletableFuture<String> future = new CompletableFuture<>();
```
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,778 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Free;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Closed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Constructed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.LeaderElectionServiceStarted;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Started;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private final Schema<ServiceUnitStateData> schema;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private final String lookupServiceAddress;
+ // TODO: define BrokerRegistry
+ private final Semaphore outstandingCleanupTombstoneMessages;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+ private final LeaderElectionService leaderElectionService;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private ScheduledFuture<?> cleanupTasks;
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+ private volatile ChannelState channelState;
+
+ enum ChannelState {
+ Closed(0),
+ Constructed(1),
+ LeaderElectionServiceStarted(2),
+ Started(3);
+
+ ChannelState(int id) {
+ this.id = id;
+ }
+ int id;
+ }
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ this.lookupServiceAddress = pulsar.getLookupServiceAddress();
+ this.schema = Schema.JSON(ServiceUnitStateData.class);
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder().build();
+ this.cleanupJobs = ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder().build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} is the leader now.", lookupServiceAddress);
+ // TODO: schedule monitorOwnerships by brokerRegistry
+ } else {
+ log.debug("This broker:{} is a follower now.", lookupServiceAddress);
+ // TODO: cancel scheduled monitorOwnerships if any
+ }
+ });
+ this.channelState = Constructed;
+ }
+
+ public synchronized void start() throws PulsarServerException {
+ validateChannelState(LeaderElectionServiceStarted, false);
+ try {
+ leaderElectionService.start();
+ this.channelState = LeaderElectionServiceStarted;
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(schema)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(schema)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ channelState = Started;
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public synchronized void close() throws PulsarServerException {
+ channelState = Closed;
+ try {
+ leaderElectionService.close();
+ log.debug("Successfully closed the channel leader election service.");
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ private void validateChannelState(ChannelState targetState, boolean checkLowerIds) {
+ int order = checkLowerIds ? -1 : 1;
+ if (Integer.compare(channelState.id, targetState.id) * order > 0) {
+ throw new IllegalStateException("Invalid channel state:" + channelState.name());
+ }
+ }
+
+ public synchronized void scheduleCompaction() throws PulsarServerException {
+ try {
+ Long threshold = pulsar.getAdminClient().topicPolicies()
+ .getCompactionThreshold(TOPIC);
+ if (threshold == null || threshold == 0) {
+ pulsar.getAdminClient().topicPolicies()
+ .setCompactionThreshold(TOPIC, COMPACTION_THRESHOLD);
+ log.info("Scheduled compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ } else {
+ log.info("Already set compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ }
+ } catch (PulsarAdminException e) {
+ throw new PulsarServerException("Failed to schedule compaction.", e);
+ }
+ }
+
+ public String getChannelOwner() {
+ validateChannelState(LeaderElectionServiceStarted, true);
+ CompletableFuture<Optional<LeaderBroker>> future = leaderElectionService.readCurrentLeader();
+ if (!future.isDone()) {
+ return null;
+ }
+ Optional<LeaderBroker> leader = future.join();
+ if (leader.isEmpty()) {
+ return null;
+ }
+ //expecting http://broker-xyz:port
+ // TODO: discard this protocol prefix removal by a util func that returns lookupServiceAddress(serviceUrl)
+ String broker = leader.get().getServiceUrl();
+ broker = broker.substring(broker.lastIndexOf('/') + 1);
+ return broker;
+ }
+
+ public boolean isChannelOwner() {
+ return isTargetBroker(getChannelOwner());
+ }
+
+ public CompletableFuture<String> getOwnerAsync(String serviceUnit) {
+ validateChannelState(Started, true);
+ ServiceUnitStateData data = tableview.get(serviceUnit);
+ if (data == null) {
+ return null;
+ }
+ switch (data.state()) {
+ case Owned, Splitting -> {
+ return CompletableFuture.completedFuture(data.broker());
+ }
+ case Assigned, Released -> {
+ return deferGetOwnerRequest(serviceUnit);
+ }
+ default -> {
+ return null;
+ }
+ }
+ }
+
+ public CompletableFuture<String> publishAssignEventAsync(String serviceUnit, String broker) {
+ CompletableFuture<String> getOwnerRequest = deferGetOwnerRequest(serviceUnit);
+ pubAsync(serviceUnit, new ServiceUnitStateData(Assigned, broker))
+ .whenComplete((__, ex) -> {
+ if (ex != null) {
+ getOwnerRequests.remove(serviceUnit);
+ if (!getOwnerRequest.isCompletedExceptionally()) {
+ getOwnerRequest.completeExceptionally(ex);
+ }
+ }
+ });
+
+ return getOwnerRequest;
+ }
+
+ public CompletableFuture<Void> publishUnloadEventAsync(Unload unload) {
+ String serviceUnit = unload.serviceUnit();
+ if (isTransferCommand(unload)) {
+ ServiceUnitStateData next = new ServiceUnitStateData(Assigned,
+ unload.destBroker().get(), unload.sourceBroker());
+ return pubAsync(serviceUnit, next).thenAccept(__ -> {
+ });
+ }
+ return tombstoneAsync(serviceUnit).thenAccept(__ -> {
+ });
+ }
+
+ public CompletableFuture<Void> publishSplitEventAsync(Split split) {
+ String serviceUnit = split.serviceUnit();
+ ServiceUnitStateData next = new ServiceUnitStateData(Splitting, split.sourceBroker());
+ return pubAsync(serviceUnit, next).thenAccept(__ -> {
+ });
+ }
+
+ private void handle(String serviceUnit, ServiceUnitStateData data) {
+ if (log.isDebugEnabled()) {
+ log.info("{} received a handle request for serviceUnit:{}, data:{}",
+ lookupServiceAddress, serviceUnit, data);
+ }
+
+ ServiceUnitState state = data == null ? Free : data.state();
+
+ // TODO : Add state validation in tableview by the compaction strategy
+ switch (state) {
+ case Owned -> handleOwnEvent(serviceUnit, data);
+ case Assigned -> handleAssignEvent(serviceUnit, data);
+ case Released -> handleReleaseEvent(serviceUnit, data);
+ case Splitting -> handleSplitEvent(serviceUnit, data);
+ case Free -> handleFreeEvent(serviceUnit);
+ default -> throw new IllegalStateException("Failed to handle channel data:" + data);
+ }
+ }
+
+ private static boolean isTransferCommand(ServiceUnitStateData data) {
+ if (data == null) {
+ return false;
+ }
+ return StringUtils.isNotEmpty(data.sourceBroker());
+ }
+
+ private static boolean isTransferCommand(Unload data) {
+ return data.destBroker().isPresent();
+ }
+
+ private static String getLogEventTag(ServiceUnitStateData data) {
+ return data == null ? "Free" :
+ isTransferCommand(data) ? "Transfer:" + data.state() : data.state().toString();
+ }
+
+ private void log(Throwable e, String serviceUnit, ServiceUnitStateData data, ServiceUnitStateData next) {
+ if (e == null) {
+ if (log.isDebugEnabled() || isTransferCommand(data)) {
+ log.info("{} handled {} event for serviceUnit:{}, cur:{}, next:{}",
+ lookupServiceAddress, getLogEventTag(data), serviceUnit,
+ data == null ? "" : data,
+ next == null ? "" : next);
+ }
+ } else {
+ log.error("{} failed to handle {} event for serviceUnit:{}, cur:{}, next:{}",
+ lookupServiceAddress, getLogEventTag(data), serviceUnit,
+ data == null ? "" : data,
+ next == null ? "" : next,
+ e);
+ }
+ }
+
+ private void handleOwnEvent(String serviceUnit, ServiceUnitStateData data) {
+ var getOwnerRequest = getOwnerRequests.remove(serviceUnit);
+ if (getOwnerRequest != null) {
+ getOwnerRequest.complete(data.broker());
+ }
+ if (isTargetBroker(data.broker())) {
+ log(null, serviceUnit, data, null);
+ }
+
+ }
+
+ private void handleAssignEvent(String serviceUnit, ServiceUnitStateData data) {
+ deferGetOwnerRequest(serviceUnit);
+ if (isTargetBroker(data.broker())) {
+ ServiceUnitStateData next = new ServiceUnitStateData(
+ isTransferCommand(data) ? Released : Owned, data.broker(), data.sourceBroker());
+ pubAsync(serviceUnit, next)
+ .whenComplete((__, e) -> log(e, serviceUnit, data, next));
+ }
+ }
+
+ private void handleReleaseEvent(String serviceUnit, ServiceUnitStateData data) {
+
+ if (isTargetBroker(data.sourceBroker())) {
+ ServiceUnitStateData next = new ServiceUnitStateData(Owned, data.broker(), data.sourceBroker());
+ // TODO: when close, pass message to clients to connect to the new broker
+ closeServiceUnit(serviceUnit)
+ .thenCompose(__ -> pubAsync(serviceUnit, next))
+ .whenComplete((__, e) -> log(e, serviceUnit, data, next));
+ }
+ }
+
+ private void handleSplitEvent(String serviceUnit, ServiceUnitStateData data) {
+ if (isTargetBroker(data.broker())) {
+ splitServiceUnit(serviceUnit)
+ .thenCompose(__ -> tombstoneAsync(serviceUnit))
+ .whenComplete((__, e) -> log(e, serviceUnit, data, null));
+ }
+ }
+
+ private void handleFreeEvent(String serviceUnit) {
+ closeServiceUnit(serviceUnit)
+ .thenAccept(__ -> {
+ var request = getOwnerRequests.remove(serviceUnit);
+ if (request != null) {
+ request.completeExceptionally(new IllegalStateException("The ownership has been unloaded. "
+ + "No owner is found for serviceUnit: " + serviceUnit));
+ }
+ })
+ .whenComplete((__, e) -> log(e, serviceUnit, null, null));
+ }
+
+ private CompletableFuture<MessageId> pubAsync(String serviceUnit, ServiceUnitStateData data) {
+ validateChannelState(Started, true);
+ CompletableFuture<MessageId> future = new CompletableFuture<>();
+ producer.newMessage()
+ .key(serviceUnit)
+ .value(data)
+ .sendAsync()
+ .whenComplete((messageId, e) -> {
+ if (e != null) {
+ log.error("Failed to publish the message: serviceUnit:{}, data:{}",
+ serviceUnit, data, e);
+ future.completeExceptionally(e);
+ } else {
+ future.complete(messageId);
+ }
+ });
+ return future;
+ }
+
+ private CompletableFuture<MessageId> tombstoneAsync(String serviceUnit) {
+ return pubAsync(serviceUnit, null);
+ }
+
+ private boolean isTargetBroker(String broker) {
+ if (broker == null) {
+ return false;
+ }
+ return broker.equals(lookupServiceAddress);
+ }
+
+ private NamespaceBundle getNamespaceBundle(String bundle) {
+ final String namespaceName = LoadManagerShared.getNamespaceNameFromBundleName(bundle);
+ final String bundleRange = LoadManagerShared.getBundleRangeFromBundleName(bundle);
+ return pulsar.getNamespaceService().getNamespaceBundleFactory().getBundle(namespaceName, bundleRange);
+ }
+
+ private CompletableFuture<String> deferGetOwnerRequest(String serviceUnit) {
+ return getOwnerRequests
+ .computeIfAbsent(serviceUnit, k -> {
+ CompletableFuture future = new CompletableFuture<>();
+ future.orTimeout(inFlightStateWaitingTimeInMillis, TimeUnit.MILLISECONDS)
+ .whenComplete((v, e) -> {
+ if (e != null) {
+ getOwnerRequests.remove(serviceUnit);
Review Comment:
```suggestion
getOwnerRequests.remove(serviceUnit, future);
```
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,778 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Free;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Closed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Constructed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.LeaderElectionServiceStarted;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Started;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private final Schema<ServiceUnitStateData> schema;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private final String lookupServiceAddress;
+ // TODO: define BrokerRegistry
+ private final Semaphore outstandingCleanupTombstoneMessages;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+ private final LeaderElectionService leaderElectionService;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private ScheduledFuture<?> cleanupTasks;
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+ private volatile ChannelState channelState;
+
+ enum ChannelState {
+ Closed(0),
+ Constructed(1),
+ LeaderElectionServiceStarted(2),
+ Started(3);
+
+ ChannelState(int id) {
+ this.id = id;
+ }
+ int id;
+ }
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ this.lookupServiceAddress = pulsar.getLookupServiceAddress();
+ this.schema = Schema.JSON(ServiceUnitStateData.class);
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder().build();
+ this.cleanupJobs = ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder().build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} is the leader now.", lookupServiceAddress);
+ // TODO: schedule monitorOwnerships by brokerRegistry
+ } else {
+ log.debug("This broker:{} is a follower now.", lookupServiceAddress);
+ // TODO: cancel scheduled monitorOwnerships if any
+ }
+ });
+ this.channelState = Constructed;
+ }
+
+ public synchronized void start() throws PulsarServerException {
Review Comment:
When starting the broker, do we need to call `start()`?
If yes, whether the first broker will run into a deadlock due to no broker available for handling new producers and consumers?
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,778 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Free;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Closed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Constructed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.LeaderElectionServiceStarted;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Started;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private final Schema<ServiceUnitStateData> schema;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private final String lookupServiceAddress;
+ // TODO: define BrokerRegistry
+ private final Semaphore outstandingCleanupTombstoneMessages;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+ private final LeaderElectionService leaderElectionService;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private ScheduledFuture<?> cleanupTasks;
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+ private volatile ChannelState channelState;
+
+ enum ChannelState {
+ Closed(0),
+ Constructed(1),
+ LeaderElectionServiceStarted(2),
+ Started(3);
+
+ ChannelState(int id) {
+ this.id = id;
+ }
+ int id;
+ }
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ this.lookupServiceAddress = pulsar.getLookupServiceAddress();
+ this.schema = Schema.JSON(ServiceUnitStateData.class);
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder().build();
+ this.cleanupJobs = ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder().build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} is the leader now.", lookupServiceAddress);
+ // TODO: schedule monitorOwnerships by brokerRegistry
+ } else {
+ log.debug("This broker:{} is a follower now.", lookupServiceAddress);
+ // TODO: cancel scheduled monitorOwnerships if any
+ }
+ });
+ this.channelState = Constructed;
+ }
+
+ public synchronized void start() throws PulsarServerException {
+ validateChannelState(LeaderElectionServiceStarted, false);
+ try {
+ leaderElectionService.start();
+ this.channelState = LeaderElectionServiceStarted;
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(schema)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(schema)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ channelState = Started;
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public synchronized void close() throws PulsarServerException {
+ channelState = Closed;
+ try {
+ leaderElectionService.close();
+ log.debug("Successfully closed the channel leader election service.");
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ private void validateChannelState(ChannelState targetState, boolean checkLowerIds) {
+ int order = checkLowerIds ? -1 : 1;
+ if (Integer.compare(channelState.id, targetState.id) * order > 0) {
+ throw new IllegalStateException("Invalid channel state:" + channelState.name());
+ }
+ }
+
+ public synchronized void scheduleCompaction() throws PulsarServerException {
+ try {
+ Long threshold = pulsar.getAdminClient().topicPolicies()
+ .getCompactionThreshold(TOPIC);
+ if (threshold == null || threshold == 0) {
+ pulsar.getAdminClient().topicPolicies()
+ .setCompactionThreshold(TOPIC, COMPACTION_THRESHOLD);
+ log.info("Scheduled compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ } else {
+ log.info("Already set compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ }
+ } catch (PulsarAdminException e) {
+ throw new PulsarServerException("Failed to schedule compaction.", e);
+ }
+ }
+
+ public String getChannelOwner() {
+ validateChannelState(LeaderElectionServiceStarted, true);
+ CompletableFuture<Optional<LeaderBroker>> future = leaderElectionService.readCurrentLeader();
+ if (!future.isDone()) {
+ return null;
+ }
+ Optional<LeaderBroker> leader = future.join();
+ if (leader.isEmpty()) {
+ return null;
+ }
+ //expecting http://broker-xyz:port
+ // TODO: discard this protocol prefix removal by a util func that returns lookupServiceAddress(serviceUrl)
+ String broker = leader.get().getServiceUrl();
+ broker = broker.substring(broker.lastIndexOf('/') + 1);
+ return broker;
+ }
+
+ public boolean isChannelOwner() {
+ return isTargetBroker(getChannelOwner());
+ }
+
+ public CompletableFuture<String> getOwnerAsync(String serviceUnit) {
+ validateChannelState(Started, true);
+ ServiceUnitStateData data = tableview.get(serviceUnit);
+ if (data == null) {
+ return null;
+ }
+ switch (data.state()) {
+ case Owned, Splitting -> {
+ return CompletableFuture.completedFuture(data.broker());
+ }
+ case Assigned, Released -> {
+ return deferGetOwnerRequest(serviceUnit);
+ }
+ default -> {
+ return null;
+ }
+ }
+ }
+
+ public CompletableFuture<String> publishAssignEventAsync(String serviceUnit, String broker) {
+ CompletableFuture<String> getOwnerRequest = deferGetOwnerRequest(serviceUnit);
+ pubAsync(serviceUnit, new ServiceUnitStateData(Assigned, broker))
+ .whenComplete((__, ex) -> {
+ if (ex != null) {
+ getOwnerRequests.remove(serviceUnit);
+ if (!getOwnerRequest.isCompletedExceptionally()) {
+ getOwnerRequest.completeExceptionally(ex);
+ }
+ }
+ });
+
+ return getOwnerRequest;
+ }
+
+ public CompletableFuture<Void> publishUnloadEventAsync(Unload unload) {
+ String serviceUnit = unload.serviceUnit();
+ if (isTransferCommand(unload)) {
+ ServiceUnitStateData next = new ServiceUnitStateData(Assigned,
+ unload.destBroker().get(), unload.sourceBroker());
+ return pubAsync(serviceUnit, next).thenAccept(__ -> {
+ });
+ }
+ return tombstoneAsync(serviceUnit).thenAccept(__ -> {
+ });
+ }
+
+ public CompletableFuture<Void> publishSplitEventAsync(Split split) {
+ String serviceUnit = split.serviceUnit();
+ ServiceUnitStateData next = new ServiceUnitStateData(Splitting, split.sourceBroker());
+ return pubAsync(serviceUnit, next).thenAccept(__ -> {
+ });
+ }
+
+ private void handle(String serviceUnit, ServiceUnitStateData data) {
+ if (log.isDebugEnabled()) {
+ log.info("{} received a handle request for serviceUnit:{}, data:{}",
+ lookupServiceAddress, serviceUnit, data);
+ }
+
+ ServiceUnitState state = data == null ? Free : data.state();
+
+ // TODO : Add state validation in tableview by the compaction strategy
+ switch (state) {
+ case Owned -> handleOwnEvent(serviceUnit, data);
+ case Assigned -> handleAssignEvent(serviceUnit, data);
+ case Released -> handleReleaseEvent(serviceUnit, data);
+ case Splitting -> handleSplitEvent(serviceUnit, data);
+ case Free -> handleFreeEvent(serviceUnit);
+ default -> throw new IllegalStateException("Failed to handle channel data:" + data);
+ }
+ }
+
+ private static boolean isTransferCommand(ServiceUnitStateData data) {
+ if (data == null) {
+ return false;
+ }
+ return StringUtils.isNotEmpty(data.sourceBroker());
+ }
+
+ private static boolean isTransferCommand(Unload data) {
+ return data.destBroker().isPresent();
+ }
+
+ private static String getLogEventTag(ServiceUnitStateData data) {
+ return data == null ? "Free" :
+ isTransferCommand(data) ? "Transfer:" + data.state() : data.state().toString();
+ }
+
+ private void log(Throwable e, String serviceUnit, ServiceUnitStateData data, ServiceUnitStateData next) {
+ if (e == null) {
+ if (log.isDebugEnabled() || isTransferCommand(data)) {
+ log.info("{} handled {} event for serviceUnit:{}, cur:{}, next:{}",
+ lookupServiceAddress, getLogEventTag(data), serviceUnit,
+ data == null ? "" : data,
+ next == null ? "" : next);
+ }
+ } else {
+ log.error("{} failed to handle {} event for serviceUnit:{}, cur:{}, next:{}",
+ lookupServiceAddress, getLogEventTag(data), serviceUnit,
+ data == null ? "" : data,
+ next == null ? "" : next,
+ e);
+ }
+ }
+
+ private void handleOwnEvent(String serviceUnit, ServiceUnitStateData data) {
+ var getOwnerRequest = getOwnerRequests.remove(serviceUnit);
+ if (getOwnerRequest != null) {
+ getOwnerRequest.complete(data.broker());
+ }
+ if (isTargetBroker(data.broker())) {
+ log(null, serviceUnit, data, null);
+ }
+
+ }
+
+ private void handleAssignEvent(String serviceUnit, ServiceUnitStateData data) {
+ deferGetOwnerRequest(serviceUnit);
+ if (isTargetBroker(data.broker())) {
+ ServiceUnitStateData next = new ServiceUnitStateData(
+ isTransferCommand(data) ? Released : Owned, data.broker(), data.sourceBroker());
+ pubAsync(serviceUnit, next)
+ .whenComplete((__, e) -> log(e, serviceUnit, data, next));
+ }
+ }
+
+ private void handleReleaseEvent(String serviceUnit, ServiceUnitStateData data) {
+
+ if (isTargetBroker(data.sourceBroker())) {
+ ServiceUnitStateData next = new ServiceUnitStateData(Owned, data.broker(), data.sourceBroker());
+ // TODO: when close, pass message to clients to connect to the new broker
+ closeServiceUnit(serviceUnit)
+ .thenCompose(__ -> pubAsync(serviceUnit, next))
+ .whenComplete((__, e) -> log(e, serviceUnit, data, next));
+ }
+ }
+
+ private void handleSplitEvent(String serviceUnit, ServiceUnitStateData data) {
+ if (isTargetBroker(data.broker())) {
+ splitServiceUnit(serviceUnit)
+ .thenCompose(__ -> tombstoneAsync(serviceUnit))
+ .whenComplete((__, e) -> log(e, serviceUnit, data, null));
+ }
+ }
+
+ private void handleFreeEvent(String serviceUnit) {
+ closeServiceUnit(serviceUnit)
+ .thenAccept(__ -> {
+ var request = getOwnerRequests.remove(serviceUnit);
+ if (request != null) {
+ request.completeExceptionally(new IllegalStateException("The ownership has been unloaded. "
+ + "No owner is found for serviceUnit: " + serviceUnit));
+ }
+ })
+ .whenComplete((__, e) -> log(e, serviceUnit, null, null));
+ }
+
+ private CompletableFuture<MessageId> pubAsync(String serviceUnit, ServiceUnitStateData data) {
+ validateChannelState(Started, true);
+ CompletableFuture<MessageId> future = new CompletableFuture<>();
+ producer.newMessage()
+ .key(serviceUnit)
+ .value(data)
+ .sendAsync()
+ .whenComplete((messageId, e) -> {
+ if (e != null) {
+ log.error("Failed to publish the message: serviceUnit:{}, data:{}",
+ serviceUnit, data, e);
+ future.completeExceptionally(e);
+ } else {
+ future.complete(messageId);
+ }
+ });
+ return future;
+ }
+
+ private CompletableFuture<MessageId> tombstoneAsync(String serviceUnit) {
+ return pubAsync(serviceUnit, null);
+ }
+
+ private boolean isTargetBroker(String broker) {
+ if (broker == null) {
+ return false;
+ }
+ return broker.equals(lookupServiceAddress);
+ }
+
+ private NamespaceBundle getNamespaceBundle(String bundle) {
+ final String namespaceName = LoadManagerShared.getNamespaceNameFromBundleName(bundle);
+ final String bundleRange = LoadManagerShared.getBundleRangeFromBundleName(bundle);
+ return pulsar.getNamespaceService().getNamespaceBundleFactory().getBundle(namespaceName, bundleRange);
+ }
+
+ private CompletableFuture<String> deferGetOwnerRequest(String serviceUnit) {
+ return getOwnerRequests
+ .computeIfAbsent(serviceUnit, k -> {
+ CompletableFuture future = new CompletableFuture<>();
+ future.orTimeout(inFlightStateWaitingTimeInMillis, TimeUnit.MILLISECONDS)
+ .whenComplete((v, e) -> {
+ if (e != null) {
+ getOwnerRequests.remove(serviceUnit);
+ log.warn("Failed to getOwner for serviceUnit:{}",
+ serviceUnit, e);
+ future.completeExceptionally((Throwable) e);
Review Comment:
If we can reach here, the future is completed with a timeout exception, right?
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,778 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Free;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Closed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Constructed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.LeaderElectionServiceStarted;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Started;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private final Schema<ServiceUnitStateData> schema;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private final String lookupServiceAddress;
+ // TODO: define BrokerRegistry
+ private final Semaphore outstandingCleanupTombstoneMessages;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+ private final LeaderElectionService leaderElectionService;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private ScheduledFuture<?> cleanupTasks;
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+ private volatile ChannelState channelState;
+
+ enum ChannelState {
+ Closed(0),
+ Constructed(1),
+ LeaderElectionServiceStarted(2),
+ Started(3);
+
+ ChannelState(int id) {
+ this.id = id;
+ }
+ int id;
+ }
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ this.lookupServiceAddress = pulsar.getLookupServiceAddress();
+ this.schema = Schema.JSON(ServiceUnitStateData.class);
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder().build();
+ this.cleanupJobs = ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder().build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} is the leader now.", lookupServiceAddress);
+ // TODO: schedule monitorOwnerships by brokerRegistry
+ } else {
+ log.debug("This broker:{} is a follower now.", lookupServiceAddress);
+ // TODO: cancel scheduled monitorOwnerships if any
+ }
+ });
+ this.channelState = Constructed;
+ }
+
+ public synchronized void start() throws PulsarServerException {
+ validateChannelState(LeaderElectionServiceStarted, false);
+ try {
+ leaderElectionService.start();
+ this.channelState = LeaderElectionServiceStarted;
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(schema)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(schema)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ channelState = Started;
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public synchronized void close() throws PulsarServerException {
+ channelState = Closed;
+ try {
+ leaderElectionService.close();
+ log.debug("Successfully closed the channel leader election service.");
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ private void validateChannelState(ChannelState targetState, boolean checkLowerIds) {
+ int order = checkLowerIds ? -1 : 1;
+ if (Integer.compare(channelState.id, targetState.id) * order > 0) {
+ throw new IllegalStateException("Invalid channel state:" + channelState.name());
+ }
+ }
+
+ public synchronized void scheduleCompaction() throws PulsarServerException {
+ try {
+ Long threshold = pulsar.getAdminClient().topicPolicies()
+ .getCompactionThreshold(TOPIC);
+ if (threshold == null || threshold == 0) {
+ pulsar.getAdminClient().topicPolicies()
+ .setCompactionThreshold(TOPIC, COMPACTION_THRESHOLD);
+ log.info("Scheduled compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ } else {
+ log.info("Already set compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ }
+ } catch (PulsarAdminException e) {
+ throw new PulsarServerException("Failed to schedule compaction.", e);
+ }
+ }
+
+ public String getChannelOwner() {
+ validateChannelState(LeaderElectionServiceStarted, true);
+ CompletableFuture<Optional<LeaderBroker>> future = leaderElectionService.readCurrentLeader();
+ if (!future.isDone()) {
+ return null;
+ }
+ Optional<LeaderBroker> leader = future.join();
+ if (leader.isEmpty()) {
+ return null;
+ }
+ //expecting http://broker-xyz:port
+ // TODO: discard this protocol prefix removal by a util func that returns lookupServiceAddress(serviceUrl)
+ String broker = leader.get().getServiceUrl();
+ broker = broker.substring(broker.lastIndexOf('/') + 1);
+ return broker;
+ }
+
+ public boolean isChannelOwner() {
+ return isTargetBroker(getChannelOwner());
+ }
+
+ public CompletableFuture<String> getOwnerAsync(String serviceUnit) {
+ validateChannelState(Started, true);
+ ServiceUnitStateData data = tableview.get(serviceUnit);
+ if (data == null) {
+ return null;
+ }
+ switch (data.state()) {
+ case Owned, Splitting -> {
+ return CompletableFuture.completedFuture(data.broker());
+ }
+ case Assigned, Released -> {
+ return deferGetOwnerRequest(serviceUnit);
+ }
+ default -> {
+ return null;
+ }
+ }
+ }
+
+ public CompletableFuture<String> publishAssignEventAsync(String serviceUnit, String broker) {
+ CompletableFuture<String> getOwnerRequest = deferGetOwnerRequest(serviceUnit);
+ pubAsync(serviceUnit, new ServiceUnitStateData(Assigned, broker))
+ .whenComplete((__, ex) -> {
+ if (ex != null) {
+ getOwnerRequests.remove(serviceUnit);
+ if (!getOwnerRequest.isCompletedExceptionally()) {
+ getOwnerRequest.completeExceptionally(ex);
+ }
+ }
+ });
+
+ return getOwnerRequest;
+ }
+
+ public CompletableFuture<Void> publishUnloadEventAsync(Unload unload) {
+ String serviceUnit = unload.serviceUnit();
+ if (isTransferCommand(unload)) {
+ ServiceUnitStateData next = new ServiceUnitStateData(Assigned,
+ unload.destBroker().get(), unload.sourceBroker());
+ return pubAsync(serviceUnit, next).thenAccept(__ -> {
+ });
Review Comment:
```suggestion
return pubAsync(serviceUnit, next).thenApply(__ -> null);
```
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,778 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Free;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Closed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Constructed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.LeaderElectionServiceStarted;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Started;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private final Schema<ServiceUnitStateData> schema;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private final String lookupServiceAddress;
+ // TODO: define BrokerRegistry
+ private final Semaphore outstandingCleanupTombstoneMessages;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+ private final LeaderElectionService leaderElectionService;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private ScheduledFuture<?> cleanupTasks;
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+ private volatile ChannelState channelState;
+
+ enum ChannelState {
+ Closed(0),
+ Constructed(1),
+ LeaderElectionServiceStarted(2),
+ Started(3);
+
+ ChannelState(int id) {
+ this.id = id;
+ }
+ int id;
+ }
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ this.lookupServiceAddress = pulsar.getLookupServiceAddress();
+ this.schema = Schema.JSON(ServiceUnitStateData.class);
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder().build();
+ this.cleanupJobs = ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder().build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} is the leader now.", lookupServiceAddress);
+ // TODO: schedule monitorOwnerships by brokerRegistry
+ } else {
+ log.debug("This broker:{} is a follower now.", lookupServiceAddress);
+ // TODO: cancel scheduled monitorOwnerships if any
+ }
+ });
+ this.channelState = Constructed;
+ }
+
+ public synchronized void start() throws PulsarServerException {
+ validateChannelState(LeaderElectionServiceStarted, false);
+ try {
+ leaderElectionService.start();
+ this.channelState = LeaderElectionServiceStarted;
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(schema)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(schema)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ channelState = Started;
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public synchronized void close() throws PulsarServerException {
+ channelState = Closed;
+ try {
+ leaderElectionService.close();
+ log.debug("Successfully closed the channel leader election service.");
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ private void validateChannelState(ChannelState targetState, boolean checkLowerIds) {
+ int order = checkLowerIds ? -1 : 1;
+ if (Integer.compare(channelState.id, targetState.id) * order > 0) {
+ throw new IllegalStateException("Invalid channel state:" + channelState.name());
+ }
+ }
+
+ public synchronized void scheduleCompaction() throws PulsarServerException {
+ try {
+ Long threshold = pulsar.getAdminClient().topicPolicies()
+ .getCompactionThreshold(TOPIC);
+ if (threshold == null || threshold == 0) {
+ pulsar.getAdminClient().topicPolicies()
+ .setCompactionThreshold(TOPIC, COMPACTION_THRESHOLD);
+ log.info("Scheduled compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ } else {
+ log.info("Already set compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ }
+ } catch (PulsarAdminException e) {
+ throw new PulsarServerException("Failed to schedule compaction.", e);
+ }
+ }
+
+ public String getChannelOwner() {
+ validateChannelState(LeaderElectionServiceStarted, true);
+ CompletableFuture<Optional<LeaderBroker>> future = leaderElectionService.readCurrentLeader();
+ if (!future.isDone()) {
+ return null;
+ }
+ Optional<LeaderBroker> leader = future.join();
+ if (leader.isEmpty()) {
+ return null;
+ }
+ //expecting http://broker-xyz:port
+ // TODO: discard this protocol prefix removal by a util func that returns lookupServiceAddress(serviceUrl)
+ String broker = leader.get().getServiceUrl();
+ broker = broker.substring(broker.lastIndexOf('/') + 1);
+ return broker;
+ }
+
+ public boolean isChannelOwner() {
+ return isTargetBroker(getChannelOwner());
+ }
+
+ public CompletableFuture<String> getOwnerAsync(String serviceUnit) {
+ validateChannelState(Started, true);
+ ServiceUnitStateData data = tableview.get(serviceUnit);
+ if (data == null) {
+ return null;
+ }
+ switch (data.state()) {
+ case Owned, Splitting -> {
+ return CompletableFuture.completedFuture(data.broker());
+ }
+ case Assigned, Released -> {
+ return deferGetOwnerRequest(serviceUnit);
+ }
+ default -> {
+ return null;
+ }
+ }
+ }
+
+ public CompletableFuture<String> publishAssignEventAsync(String serviceUnit, String broker) {
+ CompletableFuture<String> getOwnerRequest = deferGetOwnerRequest(serviceUnit);
+ pubAsync(serviceUnit, new ServiceUnitStateData(Assigned, broker))
+ .whenComplete((__, ex) -> {
+ if (ex != null) {
+ getOwnerRequests.remove(serviceUnit);
+ if (!getOwnerRequest.isCompletedExceptionally()) {
+ getOwnerRequest.completeExceptionally(ex);
+ }
+ }
+ });
+
+ return getOwnerRequest;
+ }
+
+ public CompletableFuture<Void> publishUnloadEventAsync(Unload unload) {
+ String serviceUnit = unload.serviceUnit();
+ if (isTransferCommand(unload)) {
+ ServiceUnitStateData next = new ServiceUnitStateData(Assigned,
+ unload.destBroker().get(), unload.sourceBroker());
+ return pubAsync(serviceUnit, next).thenAccept(__ -> {
+ });
+ }
+ return tombstoneAsync(serviceUnit).thenAccept(__ -> {
+ });
Review Comment:
```suggestion
return tombstoneAsync(serviceUnit).thenApply(__ -> null);
```
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,778 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Free;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Closed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Constructed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.LeaderElectionServiceStarted;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Started;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private final Schema<ServiceUnitStateData> schema;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private final String lookupServiceAddress;
+ // TODO: define BrokerRegistry
+ private final Semaphore outstandingCleanupTombstoneMessages;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+ private final LeaderElectionService leaderElectionService;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private ScheduledFuture<?> cleanupTasks;
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+ private volatile ChannelState channelState;
+
+ enum ChannelState {
+ Closed(0),
+ Constructed(1),
+ LeaderElectionServiceStarted(2),
+ Started(3);
+
+ ChannelState(int id) {
+ this.id = id;
+ }
+ int id;
+ }
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ this.lookupServiceAddress = pulsar.getLookupServiceAddress();
+ this.schema = Schema.JSON(ServiceUnitStateData.class);
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder().build();
+ this.cleanupJobs = ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder().build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} is the leader now.", lookupServiceAddress);
+ // TODO: schedule monitorOwnerships by brokerRegistry
+ } else {
+ log.debug("This broker:{} is a follower now.", lookupServiceAddress);
+ // TODO: cancel scheduled monitorOwnerships if any
+ }
+ });
+ this.channelState = Constructed;
+ }
+
+ public synchronized void start() throws PulsarServerException {
+ validateChannelState(LeaderElectionServiceStarted, false);
+ try {
+ leaderElectionService.start();
+ this.channelState = LeaderElectionServiceStarted;
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(schema)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(schema)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ channelState = Started;
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public synchronized void close() throws PulsarServerException {
+ channelState = Closed;
+ try {
+ leaderElectionService.close();
+ log.debug("Successfully closed the channel leader election service.");
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ private void validateChannelState(ChannelState targetState, boolean checkLowerIds) {
+ int order = checkLowerIds ? -1 : 1;
+ if (Integer.compare(channelState.id, targetState.id) * order > 0) {
+ throw new IllegalStateException("Invalid channel state:" + channelState.name());
+ }
+ }
+
+ public synchronized void scheduleCompaction() throws PulsarServerException {
+ try {
+ Long threshold = pulsar.getAdminClient().topicPolicies()
+ .getCompactionThreshold(TOPIC);
+ if (threshold == null || threshold == 0) {
+ pulsar.getAdminClient().topicPolicies()
+ .setCompactionThreshold(TOPIC, COMPACTION_THRESHOLD);
+ log.info("Scheduled compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ } else {
+ log.info("Already set compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ }
+ } catch (PulsarAdminException e) {
+ throw new PulsarServerException("Failed to schedule compaction.", e);
+ }
+ }
+
+ public String getChannelOwner() {
+ validateChannelState(LeaderElectionServiceStarted, true);
+ CompletableFuture<Optional<LeaderBroker>> future = leaderElectionService.readCurrentLeader();
+ if (!future.isDone()) {
+ return null;
+ }
+ Optional<LeaderBroker> leader = future.join();
+ if (leader.isEmpty()) {
+ return null;
+ }
+ //expecting http://broker-xyz:port
+ // TODO: discard this protocol prefix removal by a util func that returns lookupServiceAddress(serviceUrl)
+ String broker = leader.get().getServiceUrl();
+ broker = broker.substring(broker.lastIndexOf('/') + 1);
+ return broker;
+ }
+
+ public boolean isChannelOwner() {
+ return isTargetBroker(getChannelOwner());
+ }
+
+ public CompletableFuture<String> getOwnerAsync(String serviceUnit) {
+ validateChannelState(Started, true);
+ ServiceUnitStateData data = tableview.get(serviceUnit);
+ if (data == null) {
+ return null;
+ }
+ switch (data.state()) {
+ case Owned, Splitting -> {
+ return CompletableFuture.completedFuture(data.broker());
+ }
+ case Assigned, Released -> {
+ return deferGetOwnerRequest(serviceUnit);
+ }
+ default -> {
+ return null;
+ }
+ }
+ }
+
+ public CompletableFuture<String> publishAssignEventAsync(String serviceUnit, String broker) {
+ CompletableFuture<String> getOwnerRequest = deferGetOwnerRequest(serviceUnit);
+ pubAsync(serviceUnit, new ServiceUnitStateData(Assigned, broker))
+ .whenComplete((__, ex) -> {
+ if (ex != null) {
+ getOwnerRequests.remove(serviceUnit);
Review Comment:
```suggestion
getOwnerRequests.remove(serviceUnit, getOwnerRequest);
```
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,778 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Free;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Closed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Constructed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.LeaderElectionServiceStarted;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Started;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private final Schema<ServiceUnitStateData> schema;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private final String lookupServiceAddress;
+ // TODO: define BrokerRegistry
+ private final Semaphore outstandingCleanupTombstoneMessages;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+ private final LeaderElectionService leaderElectionService;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private ScheduledFuture<?> cleanupTasks;
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+ private volatile ChannelState channelState;
+
+ enum ChannelState {
+ Closed(0),
+ Constructed(1),
+ LeaderElectionServiceStarted(2),
+ Started(3);
+
+ ChannelState(int id) {
+ this.id = id;
+ }
+ int id;
+ }
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ this.lookupServiceAddress = pulsar.getLookupServiceAddress();
+ this.schema = Schema.JSON(ServiceUnitStateData.class);
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder().build();
+ this.cleanupJobs = ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder().build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} is the leader now.", lookupServiceAddress);
+ // TODO: schedule monitorOwnerships by brokerRegistry
+ } else {
+ log.debug("This broker:{} is a follower now.", lookupServiceAddress);
+ // TODO: cancel scheduled monitorOwnerships if any
+ }
+ });
+ this.channelState = Constructed;
+ }
+
+ public synchronized void start() throws PulsarServerException {
+ validateChannelState(LeaderElectionServiceStarted, false);
+ try {
+ leaderElectionService.start();
+ this.channelState = LeaderElectionServiceStarted;
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(schema)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(schema)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ channelState = Started;
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public synchronized void close() throws PulsarServerException {
+ channelState = Closed;
+ try {
+ leaderElectionService.close();
+ log.debug("Successfully closed the channel leader election service.");
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ private void validateChannelState(ChannelState targetState, boolean checkLowerIds) {
+ int order = checkLowerIds ? -1 : 1;
+ if (Integer.compare(channelState.id, targetState.id) * order > 0) {
+ throw new IllegalStateException("Invalid channel state:" + channelState.name());
+ }
+ }
+
+ public synchronized void scheduleCompaction() throws PulsarServerException {
+ try {
+ Long threshold = pulsar.getAdminClient().topicPolicies()
+ .getCompactionThreshold(TOPIC);
+ if (threshold == null || threshold == 0) {
+ pulsar.getAdminClient().topicPolicies()
+ .setCompactionThreshold(TOPIC, COMPACTION_THRESHOLD);
+ log.info("Scheduled compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ } else {
+ log.info("Already set compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ }
+ } catch (PulsarAdminException e) {
+ throw new PulsarServerException("Failed to schedule compaction.", e);
+ }
+ }
+
+ public String getChannelOwner() {
+ validateChannelState(LeaderElectionServiceStarted, true);
+ CompletableFuture<Optional<LeaderBroker>> future = leaderElectionService.readCurrentLeader();
+ if (!future.isDone()) {
+ return null;
+ }
+ Optional<LeaderBroker> leader = future.join();
+ if (leader.isEmpty()) {
+ return null;
+ }
+ //expecting http://broker-xyz:port
+ // TODO: discard this protocol prefix removal by a util func that returns lookupServiceAddress(serviceUrl)
+ String broker = leader.get().getServiceUrl();
+ broker = broker.substring(broker.lastIndexOf('/') + 1);
+ return broker;
+ }
+
+ public boolean isChannelOwner() {
+ return isTargetBroker(getChannelOwner());
+ }
+
+ public CompletableFuture<String> getOwnerAsync(String serviceUnit) {
+ validateChannelState(Started, true);
+ ServiceUnitStateData data = tableview.get(serviceUnit);
+ if (data == null) {
+ return null;
+ }
+ switch (data.state()) {
+ case Owned, Splitting -> {
+ return CompletableFuture.completedFuture(data.broker());
+ }
+ case Assigned, Released -> {
+ return deferGetOwnerRequest(serviceUnit);
+ }
+ default -> {
+ return null;
+ }
+ }
+ }
+
+ public CompletableFuture<String> publishAssignEventAsync(String serviceUnit, String broker) {
+ CompletableFuture<String> getOwnerRequest = deferGetOwnerRequest(serviceUnit);
+ pubAsync(serviceUnit, new ServiceUnitStateData(Assigned, broker))
+ .whenComplete((__, ex) -> {
+ if (ex != null) {
+ getOwnerRequests.remove(serviceUnit);
+ if (!getOwnerRequest.isCompletedExceptionally()) {
+ getOwnerRequest.completeExceptionally(ex);
+ }
+ }
+ });
+
+ return getOwnerRequest;
+ }
+
+ public CompletableFuture<Void> publishUnloadEventAsync(Unload unload) {
+ String serviceUnit = unload.serviceUnit();
+ if (isTransferCommand(unload)) {
+ ServiceUnitStateData next = new ServiceUnitStateData(Assigned,
+ unload.destBroker().get(), unload.sourceBroker());
+ return pubAsync(serviceUnit, next).thenAccept(__ -> {
+ });
+ }
+ return tombstoneAsync(serviceUnit).thenAccept(__ -> {
+ });
+ }
+
+ public CompletableFuture<Void> publishSplitEventAsync(Split split) {
+ String serviceUnit = split.serviceUnit();
+ ServiceUnitStateData next = new ServiceUnitStateData(Splitting, split.sourceBroker());
+ return pubAsync(serviceUnit, next).thenAccept(__ -> {
+ });
+ }
+
+ private void handle(String serviceUnit, ServiceUnitStateData data) {
+ if (log.isDebugEnabled()) {
+ log.info("{} received a handle request for serviceUnit:{}, data:{}",
+ lookupServiceAddress, serviceUnit, data);
+ }
+
+ ServiceUnitState state = data == null ? Free : data.state();
+
+ // TODO : Add state validation in tableview by the compaction strategy
+ switch (state) {
+ case Owned -> handleOwnEvent(serviceUnit, data);
+ case Assigned -> handleAssignEvent(serviceUnit, data);
+ case Released -> handleReleaseEvent(serviceUnit, data);
+ case Splitting -> handleSplitEvent(serviceUnit, data);
+ case Free -> handleFreeEvent(serviceUnit);
+ default -> throw new IllegalStateException("Failed to handle channel data:" + data);
+ }
+ }
+
+ private static boolean isTransferCommand(ServiceUnitStateData data) {
+ if (data == null) {
+ return false;
+ }
+ return StringUtils.isNotEmpty(data.sourceBroker());
+ }
+
+ private static boolean isTransferCommand(Unload data) {
+ return data.destBroker().isPresent();
+ }
+
+ private static String getLogEventTag(ServiceUnitStateData data) {
+ return data == null ? "Free" :
+ isTransferCommand(data) ? "Transfer:" + data.state() : data.state().toString();
+ }
+
+ private void log(Throwable e, String serviceUnit, ServiceUnitStateData data, ServiceUnitStateData next) {
+ if (e == null) {
+ if (log.isDebugEnabled() || isTransferCommand(data)) {
+ log.info("{} handled {} event for serviceUnit:{}, cur:{}, next:{}",
+ lookupServiceAddress, getLogEventTag(data), serviceUnit,
+ data == null ? "" : data,
+ next == null ? "" : next);
+ }
+ } else {
+ log.error("{} failed to handle {} event for serviceUnit:{}, cur:{}, next:{}",
+ lookupServiceAddress, getLogEventTag(data), serviceUnit,
+ data == null ? "" : data,
+ next == null ? "" : next,
+ e);
+ }
+ }
+
+ private void handleOwnEvent(String serviceUnit, ServiceUnitStateData data) {
+ var getOwnerRequest = getOwnerRequests.remove(serviceUnit);
+ if (getOwnerRequest != null) {
+ getOwnerRequest.complete(data.broker());
+ }
+ if (isTargetBroker(data.broker())) {
+ log(null, serviceUnit, data, null);
+ }
+
+ }
+
+ private void handleAssignEvent(String serviceUnit, ServiceUnitStateData data) {
+ deferGetOwnerRequest(serviceUnit);
+ if (isTargetBroker(data.broker())) {
+ ServiceUnitStateData next = new ServiceUnitStateData(
+ isTransferCommand(data) ? Released : Owned, data.broker(), data.sourceBroker());
+ pubAsync(serviceUnit, next)
+ .whenComplete((__, e) -> log(e, serviceUnit, data, next));
+ }
+ }
+
+ private void handleReleaseEvent(String serviceUnit, ServiceUnitStateData data) {
+
+ if (isTargetBroker(data.sourceBroker())) {
+ ServiceUnitStateData next = new ServiceUnitStateData(Owned, data.broker(), data.sourceBroker());
+ // TODO: when close, pass message to clients to connect to the new broker
+ closeServiceUnit(serviceUnit)
+ .thenCompose(__ -> pubAsync(serviceUnit, next))
+ .whenComplete((__, e) -> log(e, serviceUnit, data, next));
+ }
+ }
+
+ private void handleSplitEvent(String serviceUnit, ServiceUnitStateData data) {
+ if (isTargetBroker(data.broker())) {
+ splitServiceUnit(serviceUnit)
+ .thenCompose(__ -> tombstoneAsync(serviceUnit))
+ .whenComplete((__, e) -> log(e, serviceUnit, data, null));
+ }
+ }
+
+ private void handleFreeEvent(String serviceUnit) {
+ closeServiceUnit(serviceUnit)
+ .thenAccept(__ -> {
+ var request = getOwnerRequests.remove(serviceUnit);
+ if (request != null) {
+ request.completeExceptionally(new IllegalStateException("The ownership has been unloaded. "
+ + "No owner is found for serviceUnit: " + serviceUnit));
+ }
+ })
+ .whenComplete((__, e) -> log(e, serviceUnit, null, null));
+ }
+
+ private CompletableFuture<MessageId> pubAsync(String serviceUnit, ServiceUnitStateData data) {
+ validateChannelState(Started, true);
+ CompletableFuture<MessageId> future = new CompletableFuture<>();
+ producer.newMessage()
+ .key(serviceUnit)
+ .value(data)
+ .sendAsync()
+ .whenComplete((messageId, e) -> {
+ if (e != null) {
+ log.error("Failed to publish the message: serviceUnit:{}, data:{}",
+ serviceUnit, data, e);
+ future.completeExceptionally(e);
+ } else {
+ future.complete(messageId);
+ }
+ });
+ return future;
+ }
+
+ private CompletableFuture<MessageId> tombstoneAsync(String serviceUnit) {
+ return pubAsync(serviceUnit, null);
+ }
+
+ private boolean isTargetBroker(String broker) {
+ if (broker == null) {
+ return false;
+ }
+ return broker.equals(lookupServiceAddress);
+ }
+
+ private NamespaceBundle getNamespaceBundle(String bundle) {
+ final String namespaceName = LoadManagerShared.getNamespaceNameFromBundleName(bundle);
+ final String bundleRange = LoadManagerShared.getBundleRangeFromBundleName(bundle);
+ return pulsar.getNamespaceService().getNamespaceBundleFactory().getBundle(namespaceName, bundleRange);
+ }
+
+ private CompletableFuture<String> deferGetOwnerRequest(String serviceUnit) {
+ return getOwnerRequests
+ .computeIfAbsent(serviceUnit, k -> {
+ CompletableFuture future = new CompletableFuture<>();
+ future.orTimeout(inFlightStateWaitingTimeInMillis, TimeUnit.MILLISECONDS)
+ .whenComplete((v, e) -> {
+ if (e != null) {
+ getOwnerRequests.remove(serviceUnit);
+ log.warn("Failed to getOwner for serviceUnit:{}",
+ serviceUnit, e);
+ future.completeExceptionally((Throwable) e);
+ }
+ }
+ );
+ return future;
+ });
+ }
+
+ private CompletableFuture<Integer> closeServiceUnit(String serviceUnit) {
+ long startTime = System.nanoTime();
+ MutableInt unloadedTopics = new MutableInt();
+ NamespaceBundle bundle = getNamespaceBundle(serviceUnit);
+ return pulsar.getBrokerService().unloadServiceUnit(
+ bundle,
+ false,
+ pulsar.getConfig().getNamespaceBundleUnloadingTimeoutMs(),
+ TimeUnit.MILLISECONDS)
+ .handle((numUnloadedTopics, ex) -> {
+ if (ex != null) {
+ // ignore topic-close failure to unload bundle
+ log.error("Failed to close topics under bundle:{}",
+ bundle.toString(), ex);
+ } else {
+ unloadedTopics.setValue(numUnloadedTopics);
+ }
+ // clean up topics that failed to unload from the broker ownership cache
+ pulsar.getBrokerService().cleanUnloadedTopicFromCache(bundle);
+ return numUnloadedTopics;
+ })
+ .whenComplete((ignored, ex) -> {
+ double unloadBundleTime = TimeUnit.NANOSECONDS
+ .toMillis((System.nanoTime() - startTime));
+ log.info("Unloading bundle:{} with {} topics completed in {} ms",
+ bundle, unloadedTopics, unloadBundleTime, ex);
+ });
+ }
+
+ private CompletableFuture<Void> splitServiceUnit(String serviceUnit) {
+ // TODO: after the split we need to write the child ownerships to BSC instead of ZK.
+ // Use the configured NamespaceBundleSplitAlgorithm
+ long startTime = System.nanoTime();
+ return pulsar.getNamespaceService()
+ .splitAndOwnBundle(getNamespaceBundle(serviceUnit),
+ false,
+ NamespaceBundleSplitAlgorithm.RANGE_EQUALLY_DIVIDE_ALGO,
+ null)
+ .whenComplete((__, ex) -> {
+ double splitBundleTime = TimeUnit.NANOSECONDS
+ .toMillis((System.nanoTime() - startTime));
+ if (ex == null) {
+ log.info("Successfully split {} namespace-bundle in {} ms",
+ serviceUnit, splitBundleTime);
+ } else {
+ log.error("Failed to split {} namespace-bundle in {} ms",
+ serviceUnit, splitBundleTime, ex);
+ }
+ });
+ }
+
+ public void handleMetadataSessionEvent(SessionEvent e) {
+ if (e == SessionReestablished || e == SessionLost) {
+ lastMetadataSessionEvent = e;
+ lastMetadataSessionEventTimestamp = System.currentTimeMillis();
+ log.info("Received metadata session event:{} at timestamp:{}",
+ lastMetadataSessionEvent, lastMetadataSessionEventTimestamp);
+ }
+ }
+
+ public void handleBrokerRegistrationEvent(String broker, NotificationType type) {
+ if (type == NotificationType.Created) {
+ handleBrokerCreationEvent(broker);
+ } else if (type == NotificationType.Deleted) {
+ handleBrokerDeletionEvent(broker);
+ }
+ }
+
+ private MetadataState getMetadataState() {
+ long now = System.currentTimeMillis();
+ if (lastMetadataSessionEvent == SessionReestablished) {
+ if (now - lastMetadataSessionEventTimestamp > 1000 * maxCleanupDelayTimeInSecs) {
+ return Stable;
+ }
+ return Jittery;
+ }
+ return Unstable;
+ }
+
+ private void handleBrokerCreationEvent(String broker) {
+ CompletableFuture<Void> future = cleanupJobs.remove(broker);
+ if (future != null) {
+ future.cancel(false);
+ totalCleanupCancelledCnt++;
+ log.info("Successfully cancelled the ownership cleanup for broker:{}. isChannelOwner:{}."
+ + " Active cleanup job count:{}",
+ broker, isChannelOwner(), cleanupJobs.size());
+ } else {
+ log.info("Failed to cancel the ownership cleanup for broker:{}. isChannelOwner:{}."
+ + " There was no scheduled cleanup job. Active cleanup job count:{}",
+ broker, isChannelOwner(), cleanupJobs.size());
+ }
+ }
+
+ private void handleBrokerDeletionEvent(String broker) {
+ if (!isChannelOwner()) {
+ return;
+ }
+ MetadataState state = getMetadataState();
+ log.info("Handling broker:{} ownership cleanup based on metadata connection state:{}, event:{}, event_ts:{}:",
+ broker, state, lastMetadataSessionEvent, lastMetadataSessionEventTimestamp);
+ switch (state) {
+ case Stable -> scheduleCleanup(broker, minCleanupDelayTimeInSecs);
+ case Jittery -> scheduleCleanup(broker, maxCleanupDelayTimeInSecs);
+ case Unstable -> {
+ totalCleanupIgnoredCnt++;
+ log.error("MetadataState state is unstable. "
+ + "Ignoring the ownership cleanup request for the reported broker :{}", broker);
+ }
+ }
+ }
+
+ private void scheduleCleanup(String broker, long delayInSecs) {
+ cleanupJobs.computeIfAbsent(broker, k -> {
+ Executor delayed = CompletableFuture
+ .delayedExecutor(delayInSecs, TimeUnit.SECONDS, pulsar.getLoadManagerExecutor());
+ totalCleanupScheduledCnt++;
+ return CompletableFuture
+ .runAsync(() -> doCleanup(broker), delayed);
+ });
+
+ log.info("Scheduled ownership cleanup for broker:{} with delay:{} secs. Pending clean jobs:{}.",
+ broker, delayInSecs, cleanupJobs.size());
+ }
+
+
+ private void doCleanup(String broker) {
+ long startTime = System.nanoTime();
+ log.info("Started ownership cleanup for the inactive broker:{}", broker);
+ AtomicInteger serviceUnitTombstoneCnt = new AtomicInteger();
+ AtomicInteger serviceUnitTombstoneErrorCnt = new AtomicInteger();
+ for (Map.Entry<String, ServiceUnitStateData> etr : tableview.entrySet()) {
+ ServiceUnitStateData stateData = etr.getValue();
+ String serviceUnit = etr.getKey();
+ if (StringUtils.equals(broker, stateData.broker())
+ || StringUtils.equals(broker, stateData.sourceBroker())) {
+ try {
+ outstandingCleanupTombstoneMessages.acquire();
Review Comment:
I think we don't need `outstandingCleanupTombstoneMessages`. We can just set blockIfQueueFull to true for the producer. The pending queue size will act as a Semaphore
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,778 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Free;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Closed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Constructed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.LeaderElectionServiceStarted;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Started;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private final Schema<ServiceUnitStateData> schema;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private final String lookupServiceAddress;
+ // TODO: define BrokerRegistry
+ private final Semaphore outstandingCleanupTombstoneMessages;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+ private final LeaderElectionService leaderElectionService;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private ScheduledFuture<?> cleanupTasks;
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+ private volatile ChannelState channelState;
+
+ enum ChannelState {
+ Closed(0),
+ Constructed(1),
+ LeaderElectionServiceStarted(2),
+ Started(3);
+
+ ChannelState(int id) {
+ this.id = id;
+ }
+ int id;
+ }
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ this.lookupServiceAddress = pulsar.getLookupServiceAddress();
+ this.schema = Schema.JSON(ServiceUnitStateData.class);
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder().build();
+ this.cleanupJobs = ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder().build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} is the leader now.", lookupServiceAddress);
+ // TODO: schedule monitorOwnerships by brokerRegistry
+ } else {
+ log.debug("This broker:{} is a follower now.", lookupServiceAddress);
+ // TODO: cancel scheduled monitorOwnerships if any
+ }
+ });
+ this.channelState = Constructed;
+ }
+
+ public synchronized void start() throws PulsarServerException {
+ validateChannelState(LeaderElectionServiceStarted, false);
+ try {
+ leaderElectionService.start();
+ this.channelState = LeaderElectionServiceStarted;
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(schema)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(schema)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ channelState = Started;
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public synchronized void close() throws PulsarServerException {
+ channelState = Closed;
+ try {
+ leaderElectionService.close();
+ log.debug("Successfully closed the channel leader election service.");
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ private void validateChannelState(ChannelState targetState, boolean checkLowerIds) {
+ int order = checkLowerIds ? -1 : 1;
+ if (Integer.compare(channelState.id, targetState.id) * order > 0) {
+ throw new IllegalStateException("Invalid channel state:" + channelState.name());
+ }
+ }
+
+ public synchronized void scheduleCompaction() throws PulsarServerException {
Review Comment:
+1
If the namespace or top enabled the compaction task, it looks like we don't need to implement it again.
And the system topic has enabled the compaction be default
https://github.com/apache/pulsar/blob/cd85a67bd3a880732678b0dfd5f7769ad90a90ef/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/SystemTopic.java#L74-L78
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,778 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Free;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Closed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Constructed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.LeaderElectionServiceStarted;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Started;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private final Schema<ServiceUnitStateData> schema;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private final String lookupServiceAddress;
+ // TODO: define BrokerRegistry
+ private final Semaphore outstandingCleanupTombstoneMessages;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+ private final LeaderElectionService leaderElectionService;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private ScheduledFuture<?> cleanupTasks;
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+ private volatile ChannelState channelState;
+
+ enum ChannelState {
+ Closed(0),
+ Constructed(1),
+ LeaderElectionServiceStarted(2),
+ Started(3);
+
+ ChannelState(int id) {
+ this.id = id;
+ }
+ int id;
+ }
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ this.lookupServiceAddress = pulsar.getLookupServiceAddress();
+ this.schema = Schema.JSON(ServiceUnitStateData.class);
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder().build();
+ this.cleanupJobs = ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder().build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} is the leader now.", lookupServiceAddress);
+ // TODO: schedule monitorOwnerships by brokerRegistry
+ } else {
+ log.debug("This broker:{} is a follower now.", lookupServiceAddress);
+ // TODO: cancel scheduled monitorOwnerships if any
+ }
+ });
+ this.channelState = Constructed;
+ }
+
+ public synchronized void start() throws PulsarServerException {
+ validateChannelState(LeaderElectionServiceStarted, false);
+ try {
+ leaderElectionService.start();
+ this.channelState = LeaderElectionServiceStarted;
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(schema)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(schema)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ channelState = Started;
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public synchronized void close() throws PulsarServerException {
+ channelState = Closed;
+ try {
+ leaderElectionService.close();
+ log.debug("Successfully closed the channel leader election service.");
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ private void validateChannelState(ChannelState targetState, boolean checkLowerIds) {
+ int order = checkLowerIds ? -1 : 1;
+ if (Integer.compare(channelState.id, targetState.id) * order > 0) {
+ throw new IllegalStateException("Invalid channel state:" + channelState.name());
+ }
+ }
+
+ public synchronized void scheduleCompaction() throws PulsarServerException {
+ try {
+ Long threshold = pulsar.getAdminClient().topicPolicies()
+ .getCompactionThreshold(TOPIC);
+ if (threshold == null || threshold == 0) {
+ pulsar.getAdminClient().topicPolicies()
+ .setCompactionThreshold(TOPIC, COMPACTION_THRESHOLD);
+ log.info("Scheduled compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ } else {
+ log.info("Already set compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ }
+ } catch (PulsarAdminException e) {
+ throw new PulsarServerException("Failed to schedule compaction.", e);
+ }
+ }
+
+ public String getChannelOwner() {
+ validateChannelState(LeaderElectionServiceStarted, true);
+ CompletableFuture<Optional<LeaderBroker>> future = leaderElectionService.readCurrentLeader();
+ if (!future.isDone()) {
+ return null;
+ }
+ Optional<LeaderBroker> leader = future.join();
+ if (leader.isEmpty()) {
+ return null;
+ }
+ //expecting http://broker-xyz:port
+ // TODO: discard this protocol prefix removal by a util func that returns lookupServiceAddress(serviceUrl)
+ String broker = leader.get().getServiceUrl();
+ broker = broker.substring(broker.lastIndexOf('/') + 1);
+ return broker;
+ }
+
+ public boolean isChannelOwner() {
+ return isTargetBroker(getChannelOwner());
+ }
+
+ public CompletableFuture<String> getOwnerAsync(String serviceUnit) {
+ validateChannelState(Started, true);
+ ServiceUnitStateData data = tableview.get(serviceUnit);
+ if (data == null) {
+ return null;
+ }
+ switch (data.state()) {
+ case Owned, Splitting -> {
+ return CompletableFuture.completedFuture(data.broker());
+ }
+ case Assigned, Released -> {
+ return deferGetOwnerRequest(serviceUnit);
+ }
+ default -> {
+ return null;
+ }
+ }
+ }
+
+ public CompletableFuture<String> publishAssignEventAsync(String serviceUnit, String broker) {
+ CompletableFuture<String> getOwnerRequest = deferGetOwnerRequest(serviceUnit);
+ pubAsync(serviceUnit, new ServiceUnitStateData(Assigned, broker))
+ .whenComplete((__, ex) -> {
+ if (ex != null) {
+ getOwnerRequests.remove(serviceUnit);
+ if (!getOwnerRequest.isCompletedExceptionally()) {
+ getOwnerRequest.completeExceptionally(ex);
+ }
+ }
+ });
+
+ return getOwnerRequest;
+ }
+
+ public CompletableFuture<Void> publishUnloadEventAsync(Unload unload) {
+ String serviceUnit = unload.serviceUnit();
+ if (isTransferCommand(unload)) {
+ ServiceUnitStateData next = new ServiceUnitStateData(Assigned,
+ unload.destBroker().get(), unload.sourceBroker());
+ return pubAsync(serviceUnit, next).thenAccept(__ -> {
+ });
+ }
+ return tombstoneAsync(serviceUnit).thenAccept(__ -> {
+ });
+ }
+
+ public CompletableFuture<Void> publishSplitEventAsync(Split split) {
+ String serviceUnit = split.serviceUnit();
+ ServiceUnitStateData next = new ServiceUnitStateData(Splitting, split.sourceBroker());
+ return pubAsync(serviceUnit, next).thenAccept(__ -> {
+ });
+ }
+
+ private void handle(String serviceUnit, ServiceUnitStateData data) {
+ if (log.isDebugEnabled()) {
+ log.info("{} received a handle request for serviceUnit:{}, data:{}",
+ lookupServiceAddress, serviceUnit, data);
+ }
+
+ ServiceUnitState state = data == null ? Free : data.state();
+
+ // TODO : Add state validation in tableview by the compaction strategy
+ switch (state) {
+ case Owned -> handleOwnEvent(serviceUnit, data);
+ case Assigned -> handleAssignEvent(serviceUnit, data);
+ case Released -> handleReleaseEvent(serviceUnit, data);
+ case Splitting -> handleSplitEvent(serviceUnit, data);
+ case Free -> handleFreeEvent(serviceUnit);
+ default -> throw new IllegalStateException("Failed to handle channel data:" + data);
+ }
+ }
+
+ private static boolean isTransferCommand(ServiceUnitStateData data) {
+ if (data == null) {
+ return false;
+ }
+ return StringUtils.isNotEmpty(data.sourceBroker());
+ }
+
+ private static boolean isTransferCommand(Unload data) {
+ return data.destBroker().isPresent();
+ }
+
+ private static String getLogEventTag(ServiceUnitStateData data) {
+ return data == null ? "Free" :
+ isTransferCommand(data) ? "Transfer:" + data.state() : data.state().toString();
+ }
+
+ private void log(Throwable e, String serviceUnit, ServiceUnitStateData data, ServiceUnitStateData next) {
+ if (e == null) {
+ if (log.isDebugEnabled() || isTransferCommand(data)) {
+ log.info("{} handled {} event for serviceUnit:{}, cur:{}, next:{}",
+ lookupServiceAddress, getLogEventTag(data), serviceUnit,
+ data == null ? "" : data,
+ next == null ? "" : next);
+ }
+ } else {
+ log.error("{} failed to handle {} event for serviceUnit:{}, cur:{}, next:{}",
+ lookupServiceAddress, getLogEventTag(data), serviceUnit,
+ data == null ? "" : data,
+ next == null ? "" : next,
+ e);
+ }
+ }
+
+ private void handleOwnEvent(String serviceUnit, ServiceUnitStateData data) {
+ var getOwnerRequest = getOwnerRequests.remove(serviceUnit);
+ if (getOwnerRequest != null) {
+ getOwnerRequest.complete(data.broker());
+ }
+ if (isTargetBroker(data.broker())) {
+ log(null, serviceUnit, data, null);
+ }
+
+ }
+
+ private void handleAssignEvent(String serviceUnit, ServiceUnitStateData data) {
+ deferGetOwnerRequest(serviceUnit);
+ if (isTargetBroker(data.broker())) {
+ ServiceUnitStateData next = new ServiceUnitStateData(
+ isTransferCommand(data) ? Released : Owned, data.broker(), data.sourceBroker());
+ pubAsync(serviceUnit, next)
+ .whenComplete((__, e) -> log(e, serviceUnit, data, next));
+ }
+ }
+
+ private void handleReleaseEvent(String serviceUnit, ServiceUnitStateData data) {
+
+ if (isTargetBroker(data.sourceBroker())) {
+ ServiceUnitStateData next = new ServiceUnitStateData(Owned, data.broker(), data.sourceBroker());
+ // TODO: when close, pass message to clients to connect to the new broker
+ closeServiceUnit(serviceUnit)
+ .thenCompose(__ -> pubAsync(serviceUnit, next))
+ .whenComplete((__, e) -> log(e, serviceUnit, data, next));
+ }
+ }
+
+ private void handleSplitEvent(String serviceUnit, ServiceUnitStateData data) {
+ if (isTargetBroker(data.broker())) {
+ splitServiceUnit(serviceUnit)
+ .thenCompose(__ -> tombstoneAsync(serviceUnit))
+ .whenComplete((__, e) -> log(e, serviceUnit, data, null));
+ }
+ }
+
+ private void handleFreeEvent(String serviceUnit) {
+ closeServiceUnit(serviceUnit)
+ .thenAccept(__ -> {
+ var request = getOwnerRequests.remove(serviceUnit);
+ if (request != null) {
+ request.completeExceptionally(new IllegalStateException("The ownership has been unloaded. "
+ + "No owner is found for serviceUnit: " + serviceUnit));
+ }
+ })
+ .whenComplete((__, e) -> log(e, serviceUnit, null, null));
+ }
+
+ private CompletableFuture<MessageId> pubAsync(String serviceUnit, ServiceUnitStateData data) {
+ validateChannelState(Started, true);
+ CompletableFuture<MessageId> future = new CompletableFuture<>();
+ producer.newMessage()
+ .key(serviceUnit)
+ .value(data)
+ .sendAsync()
+ .whenComplete((messageId, e) -> {
+ if (e != null) {
+ log.error("Failed to publish the message: serviceUnit:{}, data:{}",
+ serviceUnit, data, e);
+ future.completeExceptionally(e);
+ } else {
+ future.complete(messageId);
+ }
+ });
+ return future;
+ }
+
+ private CompletableFuture<MessageId> tombstoneAsync(String serviceUnit) {
+ return pubAsync(serviceUnit, null);
+ }
+
+ private boolean isTargetBroker(String broker) {
+ if (broker == null) {
+ return false;
+ }
+ return broker.equals(lookupServiceAddress);
+ }
+
+ private NamespaceBundle getNamespaceBundle(String bundle) {
+ final String namespaceName = LoadManagerShared.getNamespaceNameFromBundleName(bundle);
+ final String bundleRange = LoadManagerShared.getBundleRangeFromBundleName(bundle);
+ return pulsar.getNamespaceService().getNamespaceBundleFactory().getBundle(namespaceName, bundleRange);
+ }
+
+ private CompletableFuture<String> deferGetOwnerRequest(String serviceUnit) {
+ return getOwnerRequests
+ .computeIfAbsent(serviceUnit, k -> {
+ CompletableFuture future = new CompletableFuture<>();
+ future.orTimeout(inFlightStateWaitingTimeInMillis, TimeUnit.MILLISECONDS)
+ .whenComplete((v, e) -> {
+ if (e != null) {
+ getOwnerRequests.remove(serviceUnit);
+ log.warn("Failed to getOwner for serviceUnit:{}",
+ serviceUnit, e);
+ future.completeExceptionally((Throwable) e);
+ }
+ }
+ );
+ return future;
+ });
+ }
+
+ private CompletableFuture<Integer> closeServiceUnit(String serviceUnit) {
+ long startTime = System.nanoTime();
+ MutableInt unloadedTopics = new MutableInt();
+ NamespaceBundle bundle = getNamespaceBundle(serviceUnit);
+ return pulsar.getBrokerService().unloadServiceUnit(
+ bundle,
+ false,
+ pulsar.getConfig().getNamespaceBundleUnloadingTimeoutMs(),
+ TimeUnit.MILLISECONDS)
+ .handle((numUnloadedTopics, ex) -> {
+ if (ex != null) {
+ // ignore topic-close failure to unload bundle
+ log.error("Failed to close topics under bundle:{}",
+ bundle.toString(), ex);
Review Comment:
Looks like we can move the exception handling to the whenComplete stage.
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,778 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Free;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Closed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Constructed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.LeaderElectionServiceStarted;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Started;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private final Schema<ServiceUnitStateData> schema;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private final String lookupServiceAddress;
+ // TODO: define BrokerRegistry
+ private final Semaphore outstandingCleanupTombstoneMessages;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+ private final LeaderElectionService leaderElectionService;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private ScheduledFuture<?> cleanupTasks;
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+ private volatile ChannelState channelState;
+
+ enum ChannelState {
+ Closed(0),
+ Constructed(1),
+ LeaderElectionServiceStarted(2),
+ Started(3);
+
+ ChannelState(int id) {
+ this.id = id;
+ }
+ int id;
+ }
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ this.lookupServiceAddress = pulsar.getLookupServiceAddress();
+ this.schema = Schema.JSON(ServiceUnitStateData.class);
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder().build();
+ this.cleanupJobs = ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder().build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} is the leader now.", lookupServiceAddress);
+ // TODO: schedule monitorOwnerships by brokerRegistry
+ } else {
+ log.debug("This broker:{} is a follower now.", lookupServiceAddress);
+ // TODO: cancel scheduled monitorOwnerships if any
+ }
+ });
+ this.channelState = Constructed;
+ }
+
+ public synchronized void start() throws PulsarServerException {
+ validateChannelState(LeaderElectionServiceStarted, false);
+ try {
+ leaderElectionService.start();
+ this.channelState = LeaderElectionServiceStarted;
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(schema)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(schema)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ channelState = Started;
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public synchronized void close() throws PulsarServerException {
+ channelState = Closed;
+ try {
+ leaderElectionService.close();
+ log.debug("Successfully closed the channel leader election service.");
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ private void validateChannelState(ChannelState targetState, boolean checkLowerIds) {
+ int order = checkLowerIds ? -1 : 1;
+ if (Integer.compare(channelState.id, targetState.id) * order > 0) {
+ throw new IllegalStateException("Invalid channel state:" + channelState.name());
+ }
+ }
+
+ public synchronized void scheduleCompaction() throws PulsarServerException {
+ try {
+ Long threshold = pulsar.getAdminClient().topicPolicies()
+ .getCompactionThreshold(TOPIC);
+ if (threshold == null || threshold == 0) {
+ pulsar.getAdminClient().topicPolicies()
+ .setCompactionThreshold(TOPIC, COMPACTION_THRESHOLD);
+ log.info("Scheduled compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ } else {
+ log.info("Already set compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ }
+ } catch (PulsarAdminException e) {
+ throw new PulsarServerException("Failed to schedule compaction.", e);
+ }
+ }
+
+ public String getChannelOwner() {
+ validateChannelState(LeaderElectionServiceStarted, true);
+ CompletableFuture<Optional<LeaderBroker>> future = leaderElectionService.readCurrentLeader();
+ if (!future.isDone()) {
+ return null;
+ }
+ Optional<LeaderBroker> leader = future.join();
+ if (leader.isEmpty()) {
+ return null;
+ }
+ //expecting http://broker-xyz:port
+ // TODO: discard this protocol prefix removal by a util func that returns lookupServiceAddress(serviceUrl)
+ String broker = leader.get().getServiceUrl();
+ broker = broker.substring(broker.lastIndexOf('/') + 1);
+ return broker;
+ }
+
+ public boolean isChannelOwner() {
+ return isTargetBroker(getChannelOwner());
+ }
+
+ public CompletableFuture<String> getOwnerAsync(String serviceUnit) {
+ validateChannelState(Started, true);
+ ServiceUnitStateData data = tableview.get(serviceUnit);
+ if (data == null) {
+ return null;
+ }
+ switch (data.state()) {
+ case Owned, Splitting -> {
+ return CompletableFuture.completedFuture(data.broker());
+ }
+ case Assigned, Released -> {
+ return deferGetOwnerRequest(serviceUnit);
+ }
+ default -> {
+ return null;
+ }
+ }
+ }
+
+ public CompletableFuture<String> publishAssignEventAsync(String serviceUnit, String broker) {
+ CompletableFuture<String> getOwnerRequest = deferGetOwnerRequest(serviceUnit);
+ pubAsync(serviceUnit, new ServiceUnitStateData(Assigned, broker))
+ .whenComplete((__, ex) -> {
+ if (ex != null) {
+ getOwnerRequests.remove(serviceUnit);
+ if (!getOwnerRequest.isCompletedExceptionally()) {
+ getOwnerRequest.completeExceptionally(ex);
+ }
+ }
+ });
+
+ return getOwnerRequest;
+ }
+
+ public CompletableFuture<Void> publishUnloadEventAsync(Unload unload) {
+ String serviceUnit = unload.serviceUnit();
+ if (isTransferCommand(unload)) {
+ ServiceUnitStateData next = new ServiceUnitStateData(Assigned,
+ unload.destBroker().get(), unload.sourceBroker());
+ return pubAsync(serviceUnit, next).thenAccept(__ -> {
+ });
+ }
+ return tombstoneAsync(serviceUnit).thenAccept(__ -> {
+ });
+ }
+
+ public CompletableFuture<Void> publishSplitEventAsync(Split split) {
+ String serviceUnit = split.serviceUnit();
+ ServiceUnitStateData next = new ServiceUnitStateData(Splitting, split.sourceBroker());
+ return pubAsync(serviceUnit, next).thenAccept(__ -> {
+ });
+ }
+
+ private void handle(String serviceUnit, ServiceUnitStateData data) {
+ if (log.isDebugEnabled()) {
+ log.info("{} received a handle request for serviceUnit:{}, data:{}",
+ lookupServiceAddress, serviceUnit, data);
+ }
+
+ ServiceUnitState state = data == null ? Free : data.state();
+
+ // TODO : Add state validation in tableview by the compaction strategy
+ switch (state) {
+ case Owned -> handleOwnEvent(serviceUnit, data);
+ case Assigned -> handleAssignEvent(serviceUnit, data);
+ case Released -> handleReleaseEvent(serviceUnit, data);
+ case Splitting -> handleSplitEvent(serviceUnit, data);
+ case Free -> handleFreeEvent(serviceUnit);
+ default -> throw new IllegalStateException("Failed to handle channel data:" + data);
+ }
+ }
+
+ private static boolean isTransferCommand(ServiceUnitStateData data) {
+ if (data == null) {
+ return false;
+ }
+ return StringUtils.isNotEmpty(data.sourceBroker());
+ }
+
+ private static boolean isTransferCommand(Unload data) {
+ return data.destBroker().isPresent();
+ }
+
+ private static String getLogEventTag(ServiceUnitStateData data) {
+ return data == null ? "Free" :
+ isTransferCommand(data) ? "Transfer:" + data.state() : data.state().toString();
+ }
+
+ private void log(Throwable e, String serviceUnit, ServiceUnitStateData data, ServiceUnitStateData next) {
+ if (e == null) {
+ if (log.isDebugEnabled() || isTransferCommand(data)) {
+ log.info("{} handled {} event for serviceUnit:{}, cur:{}, next:{}",
+ lookupServiceAddress, getLogEventTag(data), serviceUnit,
+ data == null ? "" : data,
+ next == null ? "" : next);
+ }
+ } else {
+ log.error("{} failed to handle {} event for serviceUnit:{}, cur:{}, next:{}",
+ lookupServiceAddress, getLogEventTag(data), serviceUnit,
+ data == null ? "" : data,
+ next == null ? "" : next,
+ e);
+ }
+ }
+
+ private void handleOwnEvent(String serviceUnit, ServiceUnitStateData data) {
+ var getOwnerRequest = getOwnerRequests.remove(serviceUnit);
+ if (getOwnerRequest != null) {
+ getOwnerRequest.complete(data.broker());
+ }
+ if (isTargetBroker(data.broker())) {
+ log(null, serviceUnit, data, null);
+ }
+
+ }
+
+ private void handleAssignEvent(String serviceUnit, ServiceUnitStateData data) {
+ deferGetOwnerRequest(serviceUnit);
+ if (isTargetBroker(data.broker())) {
+ ServiceUnitStateData next = new ServiceUnitStateData(
+ isTransferCommand(data) ? Released : Owned, data.broker(), data.sourceBroker());
+ pubAsync(serviceUnit, next)
+ .whenComplete((__, e) -> log(e, serviceUnit, data, next));
+ }
+ }
+
+ private void handleReleaseEvent(String serviceUnit, ServiceUnitStateData data) {
+
+ if (isTargetBroker(data.sourceBroker())) {
+ ServiceUnitStateData next = new ServiceUnitStateData(Owned, data.broker(), data.sourceBroker());
+ // TODO: when close, pass message to clients to connect to the new broker
+ closeServiceUnit(serviceUnit)
+ .thenCompose(__ -> pubAsync(serviceUnit, next))
+ .whenComplete((__, e) -> log(e, serviceUnit, data, next));
+ }
+ }
+
+ private void handleSplitEvent(String serviceUnit, ServiceUnitStateData data) {
+ if (isTargetBroker(data.broker())) {
+ splitServiceUnit(serviceUnit)
+ .thenCompose(__ -> tombstoneAsync(serviceUnit))
+ .whenComplete((__, e) -> log(e, serviceUnit, data, null));
+ }
+ }
+
+ private void handleFreeEvent(String serviceUnit) {
+ closeServiceUnit(serviceUnit)
+ .thenAccept(__ -> {
+ var request = getOwnerRequests.remove(serviceUnit);
+ if (request != null) {
+ request.completeExceptionally(new IllegalStateException("The ownership has been unloaded. "
+ + "No owner is found for serviceUnit: " + serviceUnit));
+ }
+ })
+ .whenComplete((__, e) -> log(e, serviceUnit, null, null));
+ }
+
+ private CompletableFuture<MessageId> pubAsync(String serviceUnit, ServiceUnitStateData data) {
+ validateChannelState(Started, true);
+ CompletableFuture<MessageId> future = new CompletableFuture<>();
+ producer.newMessage()
+ .key(serviceUnit)
+ .value(data)
+ .sendAsync()
+ .whenComplete((messageId, e) -> {
+ if (e != null) {
+ log.error("Failed to publish the message: serviceUnit:{}, data:{}",
+ serviceUnit, data, e);
+ future.completeExceptionally(e);
+ } else {
+ future.complete(messageId);
+ }
+ });
+ return future;
+ }
+
+ private CompletableFuture<MessageId> tombstoneAsync(String serviceUnit) {
+ return pubAsync(serviceUnit, null);
+ }
+
+ private boolean isTargetBroker(String broker) {
+ if (broker == null) {
+ return false;
+ }
+ return broker.equals(lookupServiceAddress);
+ }
+
+ private NamespaceBundle getNamespaceBundle(String bundle) {
+ final String namespaceName = LoadManagerShared.getNamespaceNameFromBundleName(bundle);
+ final String bundleRange = LoadManagerShared.getBundleRangeFromBundleName(bundle);
+ return pulsar.getNamespaceService().getNamespaceBundleFactory().getBundle(namespaceName, bundleRange);
+ }
+
+ private CompletableFuture<String> deferGetOwnerRequest(String serviceUnit) {
+ return getOwnerRequests
+ .computeIfAbsent(serviceUnit, k -> {
+ CompletableFuture future = new CompletableFuture<>();
+ future.orTimeout(inFlightStateWaitingTimeInMillis, TimeUnit.MILLISECONDS)
+ .whenComplete((v, e) -> {
+ if (e != null) {
+ getOwnerRequests.remove(serviceUnit);
+ log.warn("Failed to getOwner for serviceUnit:{}",
+ serviceUnit, e);
+ future.completeExceptionally((Throwable) e);
+ }
+ }
+ );
+ return future;
+ });
+ }
+
+ private CompletableFuture<Integer> closeServiceUnit(String serviceUnit) {
+ long startTime = System.nanoTime();
+ MutableInt unloadedTopics = new MutableInt();
+ NamespaceBundle bundle = getNamespaceBundle(serviceUnit);
+ return pulsar.getBrokerService().unloadServiceUnit(
+ bundle,
+ false,
+ pulsar.getConfig().getNamespaceBundleUnloadingTimeoutMs(),
+ TimeUnit.MILLISECONDS)
+ .handle((numUnloadedTopics, ex) -> {
+ if (ex != null) {
+ // ignore topic-close failure to unload bundle
+ log.error("Failed to close topics under bundle:{}",
+ bundle.toString(), ex);
+ } else {
+ unloadedTopics.setValue(numUnloadedTopics);
+ }
+ // clean up topics that failed to unload from the broker ownership cache
+ pulsar.getBrokerService().cleanUnloadedTopicFromCache(bundle);
+ return numUnloadedTopics;
+ })
+ .whenComplete((ignored, ex) -> {
+ double unloadBundleTime = TimeUnit.NANOSECONDS
+ .toMillis((System.nanoTime() - startTime));
+ log.info("Unloading bundle:{} with {} topics completed in {} ms",
+ bundle, unloadedTopics, unloadBundleTime, ex);
Review Comment:
It's better do not print the stack trace for an info log
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,778 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Free;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Closed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Constructed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.LeaderElectionServiceStarted;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Started;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private final Schema<ServiceUnitStateData> schema;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private final String lookupServiceAddress;
+ // TODO: define BrokerRegistry
+ private final Semaphore outstandingCleanupTombstoneMessages;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+ private final LeaderElectionService leaderElectionService;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private ScheduledFuture<?> cleanupTasks;
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+ private volatile ChannelState channelState;
+
+ enum ChannelState {
+ Closed(0),
+ Constructed(1),
+ LeaderElectionServiceStarted(2),
+ Started(3);
+
+ ChannelState(int id) {
+ this.id = id;
+ }
+ int id;
+ }
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ this.lookupServiceAddress = pulsar.getLookupServiceAddress();
+ this.schema = Schema.JSON(ServiceUnitStateData.class);
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder().build();
+ this.cleanupJobs = ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder().build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} is the leader now.", lookupServiceAddress);
+ // TODO: schedule monitorOwnerships by brokerRegistry
+ } else {
+ log.debug("This broker:{} is a follower now.", lookupServiceAddress);
+ // TODO: cancel scheduled monitorOwnerships if any
+ }
+ });
+ this.channelState = Constructed;
+ }
+
+ public synchronized void start() throws PulsarServerException {
+ validateChannelState(LeaderElectionServiceStarted, false);
+ try {
+ leaderElectionService.start();
+ this.channelState = LeaderElectionServiceStarted;
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(schema)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(schema)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ channelState = Started;
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public synchronized void close() throws PulsarServerException {
+ channelState = Closed;
+ try {
+ leaderElectionService.close();
+ log.debug("Successfully closed the channel leader election service.");
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ private void validateChannelState(ChannelState targetState, boolean checkLowerIds) {
+ int order = checkLowerIds ? -1 : 1;
+ if (Integer.compare(channelState.id, targetState.id) * order > 0) {
+ throw new IllegalStateException("Invalid channel state:" + channelState.name());
+ }
+ }
+
+ public synchronized void scheduleCompaction() throws PulsarServerException {
+ try {
+ Long threshold = pulsar.getAdminClient().topicPolicies()
+ .getCompactionThreshold(TOPIC);
+ if (threshold == null || threshold == 0) {
+ pulsar.getAdminClient().topicPolicies()
+ .setCompactionThreshold(TOPIC, COMPACTION_THRESHOLD);
+ log.info("Scheduled compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ } else {
+ log.info("Already set compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ }
+ } catch (PulsarAdminException e) {
+ throw new PulsarServerException("Failed to schedule compaction.", e);
+ }
+ }
+
+ public String getChannelOwner() {
+ validateChannelState(LeaderElectionServiceStarted, true);
+ CompletableFuture<Optional<LeaderBroker>> future = leaderElectionService.readCurrentLeader();
+ if (!future.isDone()) {
+ return null;
+ }
+ Optional<LeaderBroker> leader = future.join();
+ if (leader.isEmpty()) {
+ return null;
+ }
+ //expecting http://broker-xyz:port
+ // TODO: discard this protocol prefix removal by a util func that returns lookupServiceAddress(serviceUrl)
+ String broker = leader.get().getServiceUrl();
+ broker = broker.substring(broker.lastIndexOf('/') + 1);
+ return broker;
+ }
+
+ public boolean isChannelOwner() {
+ return isTargetBroker(getChannelOwner());
+ }
+
+ public CompletableFuture<String> getOwnerAsync(String serviceUnit) {
+ validateChannelState(Started, true);
+ ServiceUnitStateData data = tableview.get(serviceUnit);
+ if (data == null) {
+ return null;
+ }
+ switch (data.state()) {
+ case Owned, Splitting -> {
+ return CompletableFuture.completedFuture(data.broker());
+ }
+ case Assigned, Released -> {
+ return deferGetOwnerRequest(serviceUnit);
+ }
+ default -> {
+ return null;
+ }
+ }
+ }
+
+ public CompletableFuture<String> publishAssignEventAsync(String serviceUnit, String broker) {
+ CompletableFuture<String> getOwnerRequest = deferGetOwnerRequest(serviceUnit);
+ pubAsync(serviceUnit, new ServiceUnitStateData(Assigned, broker))
+ .whenComplete((__, ex) -> {
+ if (ex != null) {
+ getOwnerRequests.remove(serviceUnit);
+ if (!getOwnerRequest.isCompletedExceptionally()) {
+ getOwnerRequest.completeExceptionally(ex);
+ }
+ }
+ });
+
+ return getOwnerRequest;
+ }
+
+ public CompletableFuture<Void> publishUnloadEventAsync(Unload unload) {
+ String serviceUnit = unload.serviceUnit();
+ if (isTransferCommand(unload)) {
+ ServiceUnitStateData next = new ServiceUnitStateData(Assigned,
+ unload.destBroker().get(), unload.sourceBroker());
+ return pubAsync(serviceUnit, next).thenAccept(__ -> {
+ });
+ }
+ return tombstoneAsync(serviceUnit).thenAccept(__ -> {
+ });
+ }
+
+ public CompletableFuture<Void> publishSplitEventAsync(Split split) {
+ String serviceUnit = split.serviceUnit();
+ ServiceUnitStateData next = new ServiceUnitStateData(Splitting, split.sourceBroker());
+ return pubAsync(serviceUnit, next).thenAccept(__ -> {
+ });
+ }
+
+ private void handle(String serviceUnit, ServiceUnitStateData data) {
+ if (log.isDebugEnabled()) {
+ log.info("{} received a handle request for serviceUnit:{}, data:{}",
+ lookupServiceAddress, serviceUnit, data);
+ }
+
+ ServiceUnitState state = data == null ? Free : data.state();
+
+ // TODO : Add state validation in tableview by the compaction strategy
+ switch (state) {
+ case Owned -> handleOwnEvent(serviceUnit, data);
+ case Assigned -> handleAssignEvent(serviceUnit, data);
+ case Released -> handleReleaseEvent(serviceUnit, data);
+ case Splitting -> handleSplitEvent(serviceUnit, data);
+ case Free -> handleFreeEvent(serviceUnit);
+ default -> throw new IllegalStateException("Failed to handle channel data:" + data);
+ }
+ }
+
+ private static boolean isTransferCommand(ServiceUnitStateData data) {
+ if (data == null) {
+ return false;
+ }
+ return StringUtils.isNotEmpty(data.sourceBroker());
+ }
+
+ private static boolean isTransferCommand(Unload data) {
+ return data.destBroker().isPresent();
+ }
+
+ private static String getLogEventTag(ServiceUnitStateData data) {
+ return data == null ? "Free" :
+ isTransferCommand(data) ? "Transfer:" + data.state() : data.state().toString();
+ }
+
+ private void log(Throwable e, String serviceUnit, ServiceUnitStateData data, ServiceUnitStateData next) {
+ if (e == null) {
+ if (log.isDebugEnabled() || isTransferCommand(data)) {
+ log.info("{} handled {} event for serviceUnit:{}, cur:{}, next:{}",
+ lookupServiceAddress, getLogEventTag(data), serviceUnit,
+ data == null ? "" : data,
+ next == null ? "" : next);
+ }
+ } else {
+ log.error("{} failed to handle {} event for serviceUnit:{}, cur:{}, next:{}",
+ lookupServiceAddress, getLogEventTag(data), serviceUnit,
+ data == null ? "" : data,
+ next == null ? "" : next,
+ e);
+ }
+ }
+
+ private void handleOwnEvent(String serviceUnit, ServiceUnitStateData data) {
+ var getOwnerRequest = getOwnerRequests.remove(serviceUnit);
+ if (getOwnerRequest != null) {
+ getOwnerRequest.complete(data.broker());
+ }
+ if (isTargetBroker(data.broker())) {
+ log(null, serviceUnit, data, null);
+ }
+
+ }
+
+ private void handleAssignEvent(String serviceUnit, ServiceUnitStateData data) {
+ deferGetOwnerRequest(serviceUnit);
+ if (isTargetBroker(data.broker())) {
+ ServiceUnitStateData next = new ServiceUnitStateData(
+ isTransferCommand(data) ? Released : Owned, data.broker(), data.sourceBroker());
+ pubAsync(serviceUnit, next)
+ .whenComplete((__, e) -> log(e, serviceUnit, data, next));
+ }
+ }
+
+ private void handleReleaseEvent(String serviceUnit, ServiceUnitStateData data) {
+
+ if (isTargetBroker(data.sourceBroker())) {
+ ServiceUnitStateData next = new ServiceUnitStateData(Owned, data.broker(), data.sourceBroker());
+ // TODO: when close, pass message to clients to connect to the new broker
+ closeServiceUnit(serviceUnit)
+ .thenCompose(__ -> pubAsync(serviceUnit, next))
+ .whenComplete((__, e) -> log(e, serviceUnit, data, next));
+ }
+ }
+
+ private void handleSplitEvent(String serviceUnit, ServiceUnitStateData data) {
+ if (isTargetBroker(data.broker())) {
+ splitServiceUnit(serviceUnit)
+ .thenCompose(__ -> tombstoneAsync(serviceUnit))
+ .whenComplete((__, e) -> log(e, serviceUnit, data, null));
+ }
+ }
+
+ private void handleFreeEvent(String serviceUnit) {
+ closeServiceUnit(serviceUnit)
+ .thenAccept(__ -> {
+ var request = getOwnerRequests.remove(serviceUnit);
+ if (request != null) {
+ request.completeExceptionally(new IllegalStateException("The ownership has been unloaded. "
+ + "No owner is found for serviceUnit: " + serviceUnit));
+ }
+ })
+ .whenComplete((__, e) -> log(e, serviceUnit, null, null));
+ }
+
+ private CompletableFuture<MessageId> pubAsync(String serviceUnit, ServiceUnitStateData data) {
+ validateChannelState(Started, true);
+ CompletableFuture<MessageId> future = new CompletableFuture<>();
+ producer.newMessage()
+ .key(serviceUnit)
+ .value(data)
+ .sendAsync()
+ .whenComplete((messageId, e) -> {
+ if (e != null) {
+ log.error("Failed to publish the message: serviceUnit:{}, data:{}",
+ serviceUnit, data, e);
+ future.completeExceptionally(e);
+ } else {
+ future.complete(messageId);
+ }
+ });
+ return future;
+ }
+
+ private CompletableFuture<MessageId> tombstoneAsync(String serviceUnit) {
+ return pubAsync(serviceUnit, null);
+ }
+
+ private boolean isTargetBroker(String broker) {
+ if (broker == null) {
+ return false;
+ }
+ return broker.equals(lookupServiceAddress);
+ }
+
+ private NamespaceBundle getNamespaceBundle(String bundle) {
+ final String namespaceName = LoadManagerShared.getNamespaceNameFromBundleName(bundle);
+ final String bundleRange = LoadManagerShared.getBundleRangeFromBundleName(bundle);
+ return pulsar.getNamespaceService().getNamespaceBundleFactory().getBundle(namespaceName, bundleRange);
+ }
+
+ private CompletableFuture<String> deferGetOwnerRequest(String serviceUnit) {
+ return getOwnerRequests
+ .computeIfAbsent(serviceUnit, k -> {
+ CompletableFuture future = new CompletableFuture<>();
+ future.orTimeout(inFlightStateWaitingTimeInMillis, TimeUnit.MILLISECONDS)
+ .whenComplete((v, e) -> {
+ if (e != null) {
+ getOwnerRequests.remove(serviceUnit);
+ log.warn("Failed to getOwner for serviceUnit:{}",
+ serviceUnit, e);
+ future.completeExceptionally((Throwable) e);
+ }
+ }
+ );
+ return future;
+ });
+ }
+
+ private CompletableFuture<Integer> closeServiceUnit(String serviceUnit) {
+ long startTime = System.nanoTime();
+ MutableInt unloadedTopics = new MutableInt();
+ NamespaceBundle bundle = getNamespaceBundle(serviceUnit);
+ return pulsar.getBrokerService().unloadServiceUnit(
+ bundle,
+ false,
+ pulsar.getConfig().getNamespaceBundleUnloadingTimeoutMs(),
+ TimeUnit.MILLISECONDS)
+ .handle((numUnloadedTopics, ex) -> {
+ if (ex != null) {
+ // ignore topic-close failure to unload bundle
+ log.error("Failed to close topics under bundle:{}",
+ bundle.toString(), ex);
+ } else {
+ unloadedTopics.setValue(numUnloadedTopics);
+ }
+ // clean up topics that failed to unload from the broker ownership cache
+ pulsar.getBrokerService().cleanUnloadedTopicFromCache(bundle);
+ return numUnloadedTopics;
+ })
+ .whenComplete((ignored, ex) -> {
+ double unloadBundleTime = TimeUnit.NANOSECONDS
+ .toMillis((System.nanoTime() - startTime));
+ log.info("Unloading bundle:{} with {} topics completed in {} ms",
+ bundle, unloadedTopics, unloadBundleTime, ex);
+ });
+ }
+
+ private CompletableFuture<Void> splitServiceUnit(String serviceUnit) {
+ // TODO: after the split we need to write the child ownerships to BSC instead of ZK.
+ // Use the configured NamespaceBundleSplitAlgorithm
+ long startTime = System.nanoTime();
+ return pulsar.getNamespaceService()
+ .splitAndOwnBundle(getNamespaceBundle(serviceUnit),
+ false,
+ NamespaceBundleSplitAlgorithm.RANGE_EQUALLY_DIVIDE_ALGO,
Review Comment:
We should apply the user's configured bundle split algorithm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] heesung-sn commented on a diff in pull request #18489: [improve][broker] PIP-192 Added ServiceUnitStateChannelImpl
Posted by GitBox <gi...@apache.org>.
heesung-sn commented on code in PR #18489:
URL: https://github.com/apache/pulsar/pull/18489#discussion_r1034310124
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,778 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Free;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Closed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Constructed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.LeaderElectionServiceStarted;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Started;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private final Schema<ServiceUnitStateData> schema;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private final String lookupServiceAddress;
+ // TODO: define BrokerRegistry
+ private final Semaphore outstandingCleanupTombstoneMessages;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+ private final LeaderElectionService leaderElectionService;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private ScheduledFuture<?> cleanupTasks;
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+ private volatile ChannelState channelState;
+
+ enum ChannelState {
+ Closed(0),
+ Constructed(1),
+ LeaderElectionServiceStarted(2),
+ Started(3);
+
+ ChannelState(int id) {
+ this.id = id;
+ }
+ int id;
+ }
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ this.lookupServiceAddress = pulsar.getLookupServiceAddress();
+ this.schema = Schema.JSON(ServiceUnitStateData.class);
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder().build();
+ this.cleanupJobs = ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder().build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} is the leader now.", lookupServiceAddress);
+ // TODO: schedule monitorOwnerships by brokerRegistry
+ } else {
+ log.debug("This broker:{} is a follower now.", lookupServiceAddress);
+ // TODO: cancel scheduled monitorOwnerships if any
+ }
+ });
+ this.channelState = Constructed;
+ }
+
+ public synchronized void start() throws PulsarServerException {
Review Comment:
The channelOwner will own the system topic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] heesung-sn commented on a diff in pull request #18489: [improve][broker] PIP-192 Added ServiceUnitStateChannelImpl
Posted by GitBox <gi...@apache.org>.
heesung-sn commented on code in PR #18489:
URL: https://github.com/apache/pulsar/pull/18489#discussion_r1034421259
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,778 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Free;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Closed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Constructed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.LeaderElectionServiceStarted;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Started;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private final Schema<ServiceUnitStateData> schema;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private final String lookupServiceAddress;
+ // TODO: define BrokerRegistry
+ private final Semaphore outstandingCleanupTombstoneMessages;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+ private final LeaderElectionService leaderElectionService;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private ScheduledFuture<?> cleanupTasks;
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+ private volatile ChannelState channelState;
+
+ enum ChannelState {
+ Closed(0),
+ Constructed(1),
+ LeaderElectionServiceStarted(2),
+ Started(3);
+
+ ChannelState(int id) {
+ this.id = id;
+ }
+ int id;
+ }
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ this.lookupServiceAddress = pulsar.getLookupServiceAddress();
+ this.schema = Schema.JSON(ServiceUnitStateData.class);
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder().build();
+ this.cleanupJobs = ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder().build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} is the leader now.", lookupServiceAddress);
+ // TODO: schedule monitorOwnerships by brokerRegistry
+ } else {
+ log.debug("This broker:{} is a follower now.", lookupServiceAddress);
+ // TODO: cancel scheduled monitorOwnerships if any
+ }
+ });
+ this.channelState = Constructed;
+ }
+
+ public synchronized void start() throws PulsarServerException {
+ validateChannelState(LeaderElectionServiceStarted, false);
+ try {
+ leaderElectionService.start();
+ this.channelState = LeaderElectionServiceStarted;
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(schema)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(schema)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ channelState = Started;
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public synchronized void close() throws PulsarServerException {
+ channelState = Closed;
+ try {
+ leaderElectionService.close();
+ log.debug("Successfully closed the channel leader election service.");
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ private void validateChannelState(ChannelState targetState, boolean checkLowerIds) {
+ int order = checkLowerIds ? -1 : 1;
+ if (Integer.compare(channelState.id, targetState.id) * order > 0) {
+ throw new IllegalStateException("Invalid channel state:" + channelState.name());
+ }
+ }
+
+ public synchronized void scheduleCompaction() throws PulsarServerException {
+ try {
+ Long threshold = pulsar.getAdminClient().topicPolicies()
+ .getCompactionThreshold(TOPIC);
+ if (threshold == null || threshold == 0) {
+ pulsar.getAdminClient().topicPolicies()
+ .setCompactionThreshold(TOPIC, COMPACTION_THRESHOLD);
+ log.info("Scheduled compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ } else {
+ log.info("Already set compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
Review Comment:
removed this compaction schedule call as not required.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] heesung-sn commented on a diff in pull request #18489: [improve][broker] PIP-192 Added ServiceUnitStateChannelImpl
Posted by GitBox <gi...@apache.org>.
heesung-sn commented on code in PR #18489:
URL: https://github.com/apache/pulsar/pull/18489#discussion_r1034323280
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,778 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Free;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Closed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Constructed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.LeaderElectionServiceStarted;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Started;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private final Schema<ServiceUnitStateData> schema;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private final String lookupServiceAddress;
+ // TODO: define BrokerRegistry
+ private final Semaphore outstandingCleanupTombstoneMessages;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+ private final LeaderElectionService leaderElectionService;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private ScheduledFuture<?> cleanupTasks;
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+ private volatile ChannelState channelState;
+
+ enum ChannelState {
+ Closed(0),
+ Constructed(1),
+ LeaderElectionServiceStarted(2),
+ Started(3);
+
+ ChannelState(int id) {
+ this.id = id;
+ }
+ int id;
+ }
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ this.lookupServiceAddress = pulsar.getLookupServiceAddress();
+ this.schema = Schema.JSON(ServiceUnitStateData.class);
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder().build();
+ this.cleanupJobs = ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder().build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} is the leader now.", lookupServiceAddress);
+ // TODO: schedule monitorOwnerships by brokerRegistry
+ } else {
+ log.debug("This broker:{} is a follower now.", lookupServiceAddress);
+ // TODO: cancel scheduled monitorOwnerships if any
+ }
+ });
+ this.channelState = Constructed;
+ }
+
+ public synchronized void start() throws PulsarServerException {
+ validateChannelState(LeaderElectionServiceStarted, false);
+ try {
+ leaderElectionService.start();
+ this.channelState = LeaderElectionServiceStarted;
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(schema)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(schema)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ channelState = Started;
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public synchronized void close() throws PulsarServerException {
+ channelState = Closed;
+ try {
+ leaderElectionService.close();
+ log.debug("Successfully closed the channel leader election service.");
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ private void validateChannelState(ChannelState targetState, boolean checkLowerIds) {
+ int order = checkLowerIds ? -1 : 1;
+ if (Integer.compare(channelState.id, targetState.id) * order > 0) {
+ throw new IllegalStateException("Invalid channel state:" + channelState.name());
+ }
+ }
+
+ public synchronized void scheduleCompaction() throws PulsarServerException {
+ try {
+ Long threshold = pulsar.getAdminClient().topicPolicies()
+ .getCompactionThreshold(TOPIC);
+ if (threshold == null || threshold == 0) {
+ pulsar.getAdminClient().topicPolicies()
+ .setCompactionThreshold(TOPIC, COMPACTION_THRESHOLD);
+ log.info("Scheduled compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ } else {
+ log.info("Already set compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ }
+ } catch (PulsarAdminException e) {
+ throw new PulsarServerException("Failed to schedule compaction.", e);
+ }
+ }
+
+ public String getChannelOwner() {
+ validateChannelState(LeaderElectionServiceStarted, true);
+ CompletableFuture<Optional<LeaderBroker>> future = leaderElectionService.readCurrentLeader();
+ if (!future.isDone()) {
+ return null;
+ }
+ Optional<LeaderBroker> leader = future.join();
+ if (leader.isEmpty()) {
+ return null;
+ }
+ //expecting http://broker-xyz:port
+ // TODO: discard this protocol prefix removal by a util func that returns lookupServiceAddress(serviceUrl)
+ String broker = leader.get().getServiceUrl();
+ broker = broker.substring(broker.lastIndexOf('/') + 1);
+ return broker;
+ }
+
+ public boolean isChannelOwner() {
+ return isTargetBroker(getChannelOwner());
+ }
+
+ public CompletableFuture<String> getOwnerAsync(String serviceUnit) {
+ validateChannelState(Started, true);
+ ServiceUnitStateData data = tableview.get(serviceUnit);
+ if (data == null) {
+ return null;
+ }
+ switch (data.state()) {
+ case Owned, Splitting -> {
+ return CompletableFuture.completedFuture(data.broker());
+ }
+ case Assigned, Released -> {
+ return deferGetOwnerRequest(serviceUnit);
+ }
+ default -> {
+ return null;
+ }
+ }
+ }
+
+ public CompletableFuture<String> publishAssignEventAsync(String serviceUnit, String broker) {
+ CompletableFuture<String> getOwnerRequest = deferGetOwnerRequest(serviceUnit);
+ pubAsync(serviceUnit, new ServiceUnitStateData(Assigned, broker))
+ .whenComplete((__, ex) -> {
+ if (ex != null) {
+ getOwnerRequests.remove(serviceUnit);
+ if (!getOwnerRequest.isCompletedExceptionally()) {
+ getOwnerRequest.completeExceptionally(ex);
+ }
+ }
+ });
+
+ return getOwnerRequest;
+ }
+
+ public CompletableFuture<Void> publishUnloadEventAsync(Unload unload) {
+ String serviceUnit = unload.serviceUnit();
+ if (isTransferCommand(unload)) {
+ ServiceUnitStateData next = new ServiceUnitStateData(Assigned,
+ unload.destBroker().get(), unload.sourceBroker());
+ return pubAsync(serviceUnit, next).thenAccept(__ -> {
+ });
+ }
+ return tombstoneAsync(serviceUnit).thenAccept(__ -> {
+ });
+ }
+
+ public CompletableFuture<Void> publishSplitEventAsync(Split split) {
+ String serviceUnit = split.serviceUnit();
+ ServiceUnitStateData next = new ServiceUnitStateData(Splitting, split.sourceBroker());
+ return pubAsync(serviceUnit, next).thenAccept(__ -> {
+ });
+ }
+
+ private void handle(String serviceUnit, ServiceUnitStateData data) {
+ if (log.isDebugEnabled()) {
+ log.info("{} received a handle request for serviceUnit:{}, data:{}",
+ lookupServiceAddress, serviceUnit, data);
+ }
+
+ ServiceUnitState state = data == null ? Free : data.state();
+
+ // TODO : Add state validation in tableview by the compaction strategy
+ switch (state) {
+ case Owned -> handleOwnEvent(serviceUnit, data);
+ case Assigned -> handleAssignEvent(serviceUnit, data);
+ case Released -> handleReleaseEvent(serviceUnit, data);
+ case Splitting -> handleSplitEvent(serviceUnit, data);
+ case Free -> handleFreeEvent(serviceUnit);
+ default -> throw new IllegalStateException("Failed to handle channel data:" + data);
+ }
+ }
+
+ private static boolean isTransferCommand(ServiceUnitStateData data) {
+ if (data == null) {
+ return false;
+ }
+ return StringUtils.isNotEmpty(data.sourceBroker());
+ }
+
+ private static boolean isTransferCommand(Unload data) {
+ return data.destBroker().isPresent();
+ }
+
+ private static String getLogEventTag(ServiceUnitStateData data) {
+ return data == null ? "Free" :
+ isTransferCommand(data) ? "Transfer:" + data.state() : data.state().toString();
+ }
+
+ private void log(Throwable e, String serviceUnit, ServiceUnitStateData data, ServiceUnitStateData next) {
+ if (e == null) {
+ if (log.isDebugEnabled() || isTransferCommand(data)) {
+ log.info("{} handled {} event for serviceUnit:{}, cur:{}, next:{}",
+ lookupServiceAddress, getLogEventTag(data), serviceUnit,
+ data == null ? "" : data,
+ next == null ? "" : next);
+ }
+ } else {
+ log.error("{} failed to handle {} event for serviceUnit:{}, cur:{}, next:{}",
+ lookupServiceAddress, getLogEventTag(data), serviceUnit,
+ data == null ? "" : data,
+ next == null ? "" : next,
+ e);
+ }
+ }
+
+ private void handleOwnEvent(String serviceUnit, ServiceUnitStateData data) {
+ var getOwnerRequest = getOwnerRequests.remove(serviceUnit);
+ if (getOwnerRequest != null) {
+ getOwnerRequest.complete(data.broker());
+ }
+ if (isTargetBroker(data.broker())) {
+ log(null, serviceUnit, data, null);
+ }
+
+ }
+
+ private void handleAssignEvent(String serviceUnit, ServiceUnitStateData data) {
+ deferGetOwnerRequest(serviceUnit);
+ if (isTargetBroker(data.broker())) {
+ ServiceUnitStateData next = new ServiceUnitStateData(
+ isTransferCommand(data) ? Released : Owned, data.broker(), data.sourceBroker());
+ pubAsync(serviceUnit, next)
+ .whenComplete((__, e) -> log(e, serviceUnit, data, next));
+ }
+ }
+
+ private void handleReleaseEvent(String serviceUnit, ServiceUnitStateData data) {
+
+ if (isTargetBroker(data.sourceBroker())) {
+ ServiceUnitStateData next = new ServiceUnitStateData(Owned, data.broker(), data.sourceBroker());
+ // TODO: when close, pass message to clients to connect to the new broker
+ closeServiceUnit(serviceUnit)
+ .thenCompose(__ -> pubAsync(serviceUnit, next))
+ .whenComplete((__, e) -> log(e, serviceUnit, data, next));
+ }
+ }
+
+ private void handleSplitEvent(String serviceUnit, ServiceUnitStateData data) {
+ if (isTargetBroker(data.broker())) {
+ splitServiceUnit(serviceUnit)
+ .thenCompose(__ -> tombstoneAsync(serviceUnit))
+ .whenComplete((__, e) -> log(e, serviceUnit, data, null));
+ }
+ }
+
+ private void handleFreeEvent(String serviceUnit) {
+ closeServiceUnit(serviceUnit)
+ .thenAccept(__ -> {
+ var request = getOwnerRequests.remove(serviceUnit);
+ if (request != null) {
+ request.completeExceptionally(new IllegalStateException("The ownership has been unloaded. "
+ + "No owner is found for serviceUnit: " + serviceUnit));
+ }
+ })
+ .whenComplete((__, e) -> log(e, serviceUnit, null, null));
+ }
+
+ private CompletableFuture<MessageId> pubAsync(String serviceUnit, ServiceUnitStateData data) {
+ validateChannelState(Started, true);
+ CompletableFuture<MessageId> future = new CompletableFuture<>();
+ producer.newMessage()
+ .key(serviceUnit)
+ .value(data)
+ .sendAsync()
+ .whenComplete((messageId, e) -> {
+ if (e != null) {
+ log.error("Failed to publish the message: serviceUnit:{}, data:{}",
+ serviceUnit, data, e);
+ future.completeExceptionally(e);
+ } else {
+ future.complete(messageId);
+ }
+ });
+ return future;
+ }
+
+ private CompletableFuture<MessageId> tombstoneAsync(String serviceUnit) {
+ return pubAsync(serviceUnit, null);
+ }
+
+ private boolean isTargetBroker(String broker) {
+ if (broker == null) {
+ return false;
+ }
+ return broker.equals(lookupServiceAddress);
+ }
+
+ private NamespaceBundle getNamespaceBundle(String bundle) {
+ final String namespaceName = LoadManagerShared.getNamespaceNameFromBundleName(bundle);
+ final String bundleRange = LoadManagerShared.getBundleRangeFromBundleName(bundle);
+ return pulsar.getNamespaceService().getNamespaceBundleFactory().getBundle(namespaceName, bundleRange);
+ }
+
+ private CompletableFuture<String> deferGetOwnerRequest(String serviceUnit) {
+ return getOwnerRequests
+ .computeIfAbsent(serviceUnit, k -> {
+ CompletableFuture future = new CompletableFuture<>();
+ future.orTimeout(inFlightStateWaitingTimeInMillis, TimeUnit.MILLISECONDS)
+ .whenComplete((v, e) -> {
+ if (e != null) {
+ getOwnerRequests.remove(serviceUnit);
+ log.warn("Failed to getOwner for serviceUnit:{}",
+ serviceUnit, e);
+ future.completeExceptionally((Throwable) e);
Review Comment:
Yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] heesung-sn commented on a diff in pull request #18489: [improve][broker] PIP-192 Added ServiceUnitStateChannelImpl
Posted by GitBox <gi...@apache.org>.
heesung-sn commented on code in PR #18489:
URL: https://github.com/apache/pulsar/pull/18489#discussion_r1034420405
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,778 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Free;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Closed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Constructed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.LeaderElectionServiceStarted;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Started;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private final Schema<ServiceUnitStateData> schema;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private final String lookupServiceAddress;
+ // TODO: define BrokerRegistry
+ private final Semaphore outstandingCleanupTombstoneMessages;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+ private final LeaderElectionService leaderElectionService;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private ScheduledFuture<?> cleanupTasks;
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+ private volatile ChannelState channelState;
+
+ enum ChannelState {
+ Closed(0),
+ Constructed(1),
+ LeaderElectionServiceStarted(2),
+ Started(3);
+
+ ChannelState(int id) {
+ this.id = id;
+ }
+ int id;
+ }
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ this.lookupServiceAddress = pulsar.getLookupServiceAddress();
+ this.schema = Schema.JSON(ServiceUnitStateData.class);
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder().build();
+ this.cleanupJobs = ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder().build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} is the leader now.", lookupServiceAddress);
+ // TODO: schedule monitorOwnerships by brokerRegistry
+ } else {
+ log.debug("This broker:{} is a follower now.", lookupServiceAddress);
+ // TODO: cancel scheduled monitorOwnerships if any
+ }
+ });
+ this.channelState = Constructed;
+ }
+
+ public synchronized void start() throws PulsarServerException {
+ validateChannelState(LeaderElectionServiceStarted, false);
+ try {
+ leaderElectionService.start();
+ this.channelState = LeaderElectionServiceStarted;
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(schema)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(schema)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ channelState = Started;
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public synchronized void close() throws PulsarServerException {
+ channelState = Closed;
+ try {
+ leaderElectionService.close();
+ log.debug("Successfully closed the channel leader election service.");
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ private void validateChannelState(ChannelState targetState, boolean checkLowerIds) {
+ int order = checkLowerIds ? -1 : 1;
+ if (Integer.compare(channelState.id, targetState.id) * order > 0) {
+ throw new IllegalStateException("Invalid channel state:" + channelState.name());
+ }
+ }
+
+ public synchronized void scheduleCompaction() throws PulsarServerException {
+ try {
+ Long threshold = pulsar.getAdminClient().topicPolicies()
+ .getCompactionThreshold(TOPIC);
+ if (threshold == null || threshold == 0) {
+ pulsar.getAdminClient().topicPolicies()
+ .setCompactionThreshold(TOPIC, COMPACTION_THRESHOLD);
+ log.info("Scheduled compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ } else {
+ log.info("Already set compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ }
+ } catch (PulsarAdminException e) {
+ throw new PulsarServerException("Failed to schedule compaction.", e);
+ }
+ }
+
+ public String getChannelOwner() {
+ validateChannelState(LeaderElectionServiceStarted, true);
+ CompletableFuture<Optional<LeaderBroker>> future = leaderElectionService.readCurrentLeader();
+ if (!future.isDone()) {
Review Comment:
As PengHui suggested, we will make this return CompletedFuture.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] heesung-sn commented on a diff in pull request #18489: [improve][broker] PIP-192 Added ServiceUnitStateChannelImpl
Posted by GitBox <gi...@apache.org>.
heesung-sn commented on code in PR #18489:
URL: https://github.com/apache/pulsar/pull/18489#discussion_r1027235537
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,776 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Closed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Constructed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.LeaderElectionServiceStarted;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Started;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ private static final Schema<ServiceUnitStateData> SCHEMA = Schema.JSON(ServiceUnitStateData.class);
Review Comment:
Updated.
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,776 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Closed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Constructed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.LeaderElectionServiceStarted;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Started;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ private static final Schema<ServiceUnitStateData> SCHEMA = Schema.JSON(ServiceUnitStateData.class);
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private final String lookupServiceAddress;
+ // TODO: define BrokerRegistry
+ private final Semaphore outstandingCleanupTombstoneMessages;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+ private LeaderElectionService leaderElectionService;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private ScheduledFuture<?> cleanupTasks;
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+ private volatile ChannelState channelState;
+
+ enum ChannelState {
+ Closed(0),
+ Constructed(1),
+ LeaderElectionServiceStarted(2),
+ Started(3);
+
+ ChannelState(int id) {
+ this.id = id;
+ }
+ int id;
+ }
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ ServiceConfiguration conf = pulsar.getConfiguration();
+ this.lookupServiceAddress = pulsar.getLookupServiceAddress();
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder().build();
+ this.cleanupJobs = ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder().build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.channelState = Constructed;
+ }
+
+ public synchronized void start() throws PulsarServerException {
+ try {
Review Comment:
Updated.
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,776 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Closed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Constructed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.LeaderElectionServiceStarted;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Started;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ private static final Schema<ServiceUnitStateData> SCHEMA = Schema.JSON(ServiceUnitStateData.class);
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private final String lookupServiceAddress;
+ // TODO: define BrokerRegistry
+ private final Semaphore outstandingCleanupTombstoneMessages;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+ private LeaderElectionService leaderElectionService;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private ScheduledFuture<?> cleanupTasks;
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+ private volatile ChannelState channelState;
+
+ enum ChannelState {
+ Closed(0),
+ Constructed(1),
+ LeaderElectionServiceStarted(2),
+ Started(3);
+
+ ChannelState(int id) {
+ this.id = id;
+ }
+ int id;
+ }
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ ServiceConfiguration conf = pulsar.getConfiguration();
+ this.lookupServiceAddress = pulsar.getLookupServiceAddress();
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder().build();
+ this.cleanupJobs = ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder().build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.channelState = Constructed;
+ }
+
+ public synchronized void start() throws PulsarServerException {
+ try {
+ if (leaderElectionService != null) {
+ leaderElectionService.close();
+ log.debug("Closed the channel leader election service.");
+ }
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} is the leader now.", lookupServiceAddress);
+ // TODO: schedule monitorOwnerships by brokerRegistry
+ } else {
+ log.debug("This broker:{} is a follower now.", lookupServiceAddress);
+ // TODO: cancel scheduled monitorOwnerships if any
+ }
+ });
+ leaderElectionService.start();
+ this.channelState = LeaderElectionServiceStarted;
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(SCHEMA)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(SCHEMA)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ channelState = Started;
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public synchronized void close() throws PulsarServerException {
+ channelState = Closed;
+ try {
+ if (leaderElectionService != null) {
+ leaderElectionService.close();
+ leaderElectionService = null;
+ log.debug("Successfully closed the channel leader election service.");
+ }
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ private void validateChannelState(ChannelState targetState) {
+ if (channelState.id < targetState.id) {
+ throw new IllegalStateException("Invalid channel state:" + channelState.name());
+ }
+ }
+
+ public synchronized void scheduleCompaction() throws PulsarServerException {
+ try {
+ Long threshold = pulsar.getAdminClient().topicPolicies()
+ .getCompactionThreshold(TOPIC);
+ if (threshold == null || threshold == 0) {
+ pulsar.getAdminClient().topicPolicies()
+ .setCompactionThreshold(TOPIC, COMPACTION_THRESHOLD);
+ log.info("Scheduled compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ } else {
+ log.info("Already set compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ }
+ } catch (PulsarAdminException e) {
+ throw new PulsarServerException("Failed to schedule compaction.", e);
+ }
+ }
+
+ public String getChannelOwner() {
+ validateChannelState(LeaderElectionServiceStarted);
+ CompletableFuture<Optional<LeaderBroker>> future = leaderElectionService.readCurrentLeader();
+ if (!future.isDone()) {
+ return null;
+ }
+ Optional<LeaderBroker> leader = future.join();
+ if (leader.isEmpty()) {
+ return null;
+ }
+ //expecting http://broker-xyz:port
+ // TODO: discard this protocol prefix removal by a util func that returns lookupServiceAddress(serviceUrl)
+ String broker = leader.get().getServiceUrl();
+ broker = broker.substring(broker.lastIndexOf('/') + 1);
+ return broker;
+ }
+
+ public boolean isChannelOwner() {
+ return isTargetBroker(getChannelOwner());
+ }
+
+ public CompletableFuture<String> getOwnerAsync(String serviceUnit) {
+ validateChannelState(Started);
+ ServiceUnitStateData data = tableview.get(serviceUnit);
+ if (data == null) {
+ return null;
+ }
+ switch (data.state()) {
+ case Owned, Splitting -> {
+ return CompletableFuture.completedFuture(data.broker());
+ }
+ case Assigned, Released -> {
+ return deferGetOwnerRequest(serviceUnit);
+ }
+ default -> {
+ return null;
+ }
+ }
+ }
+
+ public CompletableFuture<String> publishAssignEventAsync(String serviceUnit, String broker) {
+ CompletableFuture<String> getOwnerRequest = deferGetOwnerRequest(serviceUnit);
+ pubAsync(serviceUnit, new ServiceUnitStateData(Assigned, broker))
+ .whenComplete((__, ex) -> {
+ if (ex != null) {
+ getOwnerRequests.remove(serviceUnit);
+ if (!getOwnerRequest.isCompletedExceptionally()) {
+ getOwnerRequest.completeExceptionally(ex);
+ }
+ }
+ });
+
+ return getOwnerRequest;
+ }
+
+ public CompletableFuture<Void> publishUnloadEventAsync(Unload unload) {
+ String serviceUnit = unload.serviceUnit();
+ if (isTransferCommand(unload)) {
+ ServiceUnitStateData next = new ServiceUnitStateData(Assigned,
+ unload.destBroker().get(), unload.sourceBroker());
+ return pubAsync(serviceUnit, next).thenAccept(__ -> {
+ });
+ }
+ return tombstoneAsync(serviceUnit).thenAccept(__ -> {
+ });
+ }
+
+ public CompletableFuture<Void> publishSplitEventAsync(Split split) {
+ String serviceUnit = split.serviceUnit();
+ ServiceUnitStateData next = new ServiceUnitStateData(Splitting, split.sourceBroker());
+ return pubAsync(serviceUnit, next).thenAccept(__ -> {
+ });
+ }
+
+ private void handle(String serviceUnit, ServiceUnitStateData data) {
+ if (log.isDebugEnabled()) {
+ log.info("{} received a handle request for serviceUnit:{}, data:{}",
+ lookupServiceAddress, serviceUnit, data);
+ }
+
+ ServiceUnitState state = data == null ? ServiceUnitState.Free : data.state();
+
+ // TODO : Add state validation in tableview by the compaction strategy
+ switch (state) {
+ case Owned -> handleOwnEvent(serviceUnit, data);
+ case Assigned -> handleAssignEvent(serviceUnit, data);
+ case Released -> handleReleaseEvent(serviceUnit, data);
+ case Splitting -> handleSplitEvent(serviceUnit, data);
+ case Free -> handleFreeEvent(serviceUnit);
+ default -> throw new IllegalStateException("Failed to handle channel data:" + data);
+ }
+ }
+
+ private static boolean isTransferCommand(ServiceUnitStateData data) {
+ if (data == null) {
+ return false;
+ }
+ return StringUtils.isNotEmpty(data.sourceBroker());
+ }
+
+ private static boolean isTransferCommand(Unload data) {
+ return data.destBroker().isPresent();
+ }
+
+ private static String getLogEventTag(ServiceUnitStateData data) {
+ return data == null ? "Free" :
+ isTransferCommand(data) ? "Transfer:" + data.state() : data.state().toString();
+ }
+
+ private void log(Throwable e, String serviceUnit, ServiceUnitStateData data, ServiceUnitStateData next) {
+ if (e == null) {
+ if (log.isDebugEnabled() || isTransferCommand(data)) {
+ log.info("{} handled {} event for serviceUnit:{}, cur:{}, next:{}",
+ lookupServiceAddress, getLogEventTag(data), serviceUnit,
+ data == null ? "" : data,
+ next == null ? "" : next);
+ }
+ } else {
+ log.error("{} failed to handle {} event for serviceUnit:{}, cur:{}, next:{}",
+ lookupServiceAddress, getLogEventTag(data), serviceUnit,
+ data == null ? "" : data,
+ next == null ? "" : next,
+ e);
+ }
+ }
+
+ private void handleOwnEvent(String serviceUnit, ServiceUnitStateData data) {
+ var getOwnerRequest = getOwnerRequests.remove(serviceUnit);
+ if (getOwnerRequest != null) {
+ getOwnerRequest.complete(data.broker());
+ }
+ if (isTargetBroker(data.broker())) {
+ log(null, serviceUnit, data, null);
+ }
+
+ }
+
+ private void handleAssignEvent(String serviceUnit, ServiceUnitStateData data) {
+ deferGetOwnerRequest(serviceUnit);
+ if (isTargetBroker(data.broker())) {
+ ServiceUnitStateData next = new ServiceUnitStateData(
+ isTransferCommand(data) ? Released : Owned, data.broker(), data.sourceBroker());
+ pubAsync(serviceUnit, next)
+ .whenComplete((__, e) -> log(e, serviceUnit, data, next));
+ }
+ }
+
+ private void handleReleaseEvent(String serviceUnit, ServiceUnitStateData data) {
+
+ if (isTargetBroker(data.sourceBroker())) {
+ ServiceUnitStateData next = new ServiceUnitStateData(Owned, data.broker(), data.sourceBroker());
+ // TODO: when close, pass message to clients to connect to the new broker
+ closeServiceUnit(serviceUnit)
+ .thenCompose(__ -> pubAsync(serviceUnit, next))
+ .whenComplete((__, e) -> log(e, serviceUnit, data, next));
+ }
+ }
+
+ private void handleSplitEvent(String serviceUnit, ServiceUnitStateData data) {
+ if (isTargetBroker(data.broker())) {
+ splitServiceUnit(serviceUnit)
+ .thenCompose(__ -> tombstoneAsync(serviceUnit))
+ .whenComplete((__, e) -> log(e, serviceUnit, data, null));
+ }
+ }
+
+ private void handleFreeEvent(String serviceUnit) {
+ closeServiceUnit(serviceUnit)
+ .thenAccept(__ -> {
+ var request = getOwnerRequests.remove(serviceUnit);
+ if (request != null) {
+ request.completeExceptionally(new IllegalStateException("The ownership has been unloaded. "
+ + "No owner is found for serviceUnit: " + serviceUnit));
+ }
+ })
+ .whenComplete((__, e) -> log(e, serviceUnit, null, null));
+ }
+
+ private CompletableFuture<MessageId> pubAsync(String serviceUnit, ServiceUnitStateData data) {
+ validateChannelState(Started);
+ CompletableFuture<MessageId> future = new CompletableFuture<>();
+ producer.newMessage()
+ .key(serviceUnit)
+ .value(data)
+ .sendAsync()
+ .whenComplete((messageId, e) -> {
+ if (e != null) {
+ future.completeExceptionally(e);
Review Comment:
Updated.
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,776 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Closed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Constructed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.LeaderElectionServiceStarted;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Started;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ private static final Schema<ServiceUnitStateData> SCHEMA = Schema.JSON(ServiceUnitStateData.class);
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private final String lookupServiceAddress;
+ // TODO: define BrokerRegistry
+ private final Semaphore outstandingCleanupTombstoneMessages;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+ private LeaderElectionService leaderElectionService;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private ScheduledFuture<?> cleanupTasks;
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+ private volatile ChannelState channelState;
+
+ enum ChannelState {
+ Closed(0),
+ Constructed(1),
+ LeaderElectionServiceStarted(2),
+ Started(3);
+
+ ChannelState(int id) {
+ this.id = id;
+ }
+ int id;
+ }
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ ServiceConfiguration conf = pulsar.getConfiguration();
+ this.lookupServiceAddress = pulsar.getLookupServiceAddress();
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder().build();
+ this.cleanupJobs = ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder().build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.channelState = Constructed;
+ }
+
+ public synchronized void start() throws PulsarServerException {
+ try {
+ if (leaderElectionService != null) {
+ leaderElectionService.close();
+ log.debug("Closed the channel leader election service.");
+ }
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} is the leader now.", lookupServiceAddress);
+ // TODO: schedule monitorOwnerships by brokerRegistry
+ } else {
+ log.debug("This broker:{} is a follower now.", lookupServiceAddress);
+ // TODO: cancel scheduled monitorOwnerships if any
+ }
+ });
+ leaderElectionService.start();
+ this.channelState = LeaderElectionServiceStarted;
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(SCHEMA)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(SCHEMA)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ channelState = Started;
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public synchronized void close() throws PulsarServerException {
+ channelState = Closed;
+ try {
+ if (leaderElectionService != null) {
+ leaderElectionService.close();
+ leaderElectionService = null;
+ log.debug("Successfully closed the channel leader election service.");
+ }
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ private void validateChannelState(ChannelState targetState) {
+ if (channelState.id < targetState.id) {
+ throw new IllegalStateException("Invalid channel state:" + channelState.name());
+ }
+ }
+
+ public synchronized void scheduleCompaction() throws PulsarServerException {
+ try {
+ Long threshold = pulsar.getAdminClient().topicPolicies()
+ .getCompactionThreshold(TOPIC);
+ if (threshold == null || threshold == 0) {
+ pulsar.getAdminClient().topicPolicies()
+ .setCompactionThreshold(TOPIC, COMPACTION_THRESHOLD);
+ log.info("Scheduled compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ } else {
+ log.info("Already set compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ }
+ } catch (PulsarAdminException e) {
+ throw new PulsarServerException("Failed to schedule compaction.", e);
+ }
+ }
+
+ public String getChannelOwner() {
+ validateChannelState(LeaderElectionServiceStarted);
+ CompletableFuture<Optional<LeaderBroker>> future = leaderElectionService.readCurrentLeader();
+ if (!future.isDone()) {
+ return null;
+ }
+ Optional<LeaderBroker> leader = future.join();
+ if (leader.isEmpty()) {
+ return null;
+ }
+ //expecting http://broker-xyz:port
+ // TODO: discard this protocol prefix removal by a util func that returns lookupServiceAddress(serviceUrl)
+ String broker = leader.get().getServiceUrl();
+ broker = broker.substring(broker.lastIndexOf('/') + 1);
+ return broker;
+ }
+
+ public boolean isChannelOwner() {
+ return isTargetBroker(getChannelOwner());
+ }
+
+ public CompletableFuture<String> getOwnerAsync(String serviceUnit) {
+ validateChannelState(Started);
+ ServiceUnitStateData data = tableview.get(serviceUnit);
+ if (data == null) {
+ return null;
+ }
+ switch (data.state()) {
+ case Owned, Splitting -> {
+ return CompletableFuture.completedFuture(data.broker());
+ }
+ case Assigned, Released -> {
+ return deferGetOwnerRequest(serviceUnit);
+ }
+ default -> {
+ return null;
+ }
+ }
+ }
+
+ public CompletableFuture<String> publishAssignEventAsync(String serviceUnit, String broker) {
+ CompletableFuture<String> getOwnerRequest = deferGetOwnerRequest(serviceUnit);
+ pubAsync(serviceUnit, new ServiceUnitStateData(Assigned, broker))
+ .whenComplete((__, ex) -> {
+ if (ex != null) {
+ getOwnerRequests.remove(serviceUnit);
+ if (!getOwnerRequest.isCompletedExceptionally()) {
+ getOwnerRequest.completeExceptionally(ex);
+ }
+ }
+ });
+
+ return getOwnerRequest;
+ }
+
+ public CompletableFuture<Void> publishUnloadEventAsync(Unload unload) {
+ String serviceUnit = unload.serviceUnit();
+ if (isTransferCommand(unload)) {
+ ServiceUnitStateData next = new ServiceUnitStateData(Assigned,
+ unload.destBroker().get(), unload.sourceBroker());
+ return pubAsync(serviceUnit, next).thenAccept(__ -> {
+ });
+ }
+ return tombstoneAsync(serviceUnit).thenAccept(__ -> {
+ });
+ }
+
+ public CompletableFuture<Void> publishSplitEventAsync(Split split) {
+ String serviceUnit = split.serviceUnit();
+ ServiceUnitStateData next = new ServiceUnitStateData(Splitting, split.sourceBroker());
+ return pubAsync(serviceUnit, next).thenAccept(__ -> {
+ });
+ }
+
+ private void handle(String serviceUnit, ServiceUnitStateData data) {
+ if (log.isDebugEnabled()) {
+ log.info("{} received a handle request for serviceUnit:{}, data:{}",
+ lookupServiceAddress, serviceUnit, data);
+ }
+
+ ServiceUnitState state = data == null ? ServiceUnitState.Free : data.state();
+
+ // TODO : Add state validation in tableview by the compaction strategy
+ switch (state) {
+ case Owned -> handleOwnEvent(serviceUnit, data);
+ case Assigned -> handleAssignEvent(serviceUnit, data);
+ case Released -> handleReleaseEvent(serviceUnit, data);
+ case Splitting -> handleSplitEvent(serviceUnit, data);
+ case Free -> handleFreeEvent(serviceUnit);
+ default -> throw new IllegalStateException("Failed to handle channel data:" + data);
+ }
+ }
+
+ private static boolean isTransferCommand(ServiceUnitStateData data) {
+ if (data == null) {
+ return false;
+ }
+ return StringUtils.isNotEmpty(data.sourceBroker());
+ }
+
+ private static boolean isTransferCommand(Unload data) {
+ return data.destBroker().isPresent();
+ }
+
+ private static String getLogEventTag(ServiceUnitStateData data) {
+ return data == null ? "Free" :
+ isTransferCommand(data) ? "Transfer:" + data.state() : data.state().toString();
+ }
+
+ private void log(Throwable e, String serviceUnit, ServiceUnitStateData data, ServiceUnitStateData next) {
+ if (e == null) {
+ if (log.isDebugEnabled() || isTransferCommand(data)) {
+ log.info("{} handled {} event for serviceUnit:{}, cur:{}, next:{}",
+ lookupServiceAddress, getLogEventTag(data), serviceUnit,
+ data == null ? "" : data,
+ next == null ? "" : next);
+ }
+ } else {
+ log.error("{} failed to handle {} event for serviceUnit:{}, cur:{}, next:{}",
+ lookupServiceAddress, getLogEventTag(data), serviceUnit,
+ data == null ? "" : data,
+ next == null ? "" : next,
+ e);
+ }
+ }
+
+ private void handleOwnEvent(String serviceUnit, ServiceUnitStateData data) {
+ var getOwnerRequest = getOwnerRequests.remove(serviceUnit);
+ if (getOwnerRequest != null) {
+ getOwnerRequest.complete(data.broker());
+ }
+ if (isTargetBroker(data.broker())) {
+ log(null, serviceUnit, data, null);
+ }
+
+ }
+
+ private void handleAssignEvent(String serviceUnit, ServiceUnitStateData data) {
+ deferGetOwnerRequest(serviceUnit);
+ if (isTargetBroker(data.broker())) {
+ ServiceUnitStateData next = new ServiceUnitStateData(
+ isTransferCommand(data) ? Released : Owned, data.broker(), data.sourceBroker());
+ pubAsync(serviceUnit, next)
+ .whenComplete((__, e) -> log(e, serviceUnit, data, next));
+ }
+ }
+
+ private void handleReleaseEvent(String serviceUnit, ServiceUnitStateData data) {
+
+ if (isTargetBroker(data.sourceBroker())) {
+ ServiceUnitStateData next = new ServiceUnitStateData(Owned, data.broker(), data.sourceBroker());
+ // TODO: when close, pass message to clients to connect to the new broker
+ closeServiceUnit(serviceUnit)
+ .thenCompose(__ -> pubAsync(serviceUnit, next))
+ .whenComplete((__, e) -> log(e, serviceUnit, data, next));
+ }
+ }
+
+ private void handleSplitEvent(String serviceUnit, ServiceUnitStateData data) {
+ if (isTargetBroker(data.broker())) {
+ splitServiceUnit(serviceUnit)
+ .thenCompose(__ -> tombstoneAsync(serviceUnit))
+ .whenComplete((__, e) -> log(e, serviceUnit, data, null));
+ }
+ }
+
+ private void handleFreeEvent(String serviceUnit) {
+ closeServiceUnit(serviceUnit)
+ .thenAccept(__ -> {
+ var request = getOwnerRequests.remove(serviceUnit);
+ if (request != null) {
+ request.completeExceptionally(new IllegalStateException("The ownership has been unloaded. "
+ + "No owner is found for serviceUnit: " + serviceUnit));
+ }
+ })
+ .whenComplete((__, e) -> log(e, serviceUnit, null, null));
+ }
+
+ private CompletableFuture<MessageId> pubAsync(String serviceUnit, ServiceUnitStateData data) {
+ validateChannelState(Started);
+ CompletableFuture<MessageId> future = new CompletableFuture<>();
+ producer.newMessage()
+ .key(serviceUnit)
+ .value(data)
+ .sendAsync()
+ .whenComplete((messageId, e) -> {
+ if (e != null) {
+ future.completeExceptionally(e);
+ } else {
+ future.complete(messageId);
+ }
+ });
+ return future;
+ }
+
+ private CompletableFuture<MessageId> tombstoneAsync(String serviceUnit) {
+ return pubAsync(serviceUnit, null);
+ }
+
+ private boolean isTargetBroker(String broker) {
+ if (broker == null) {
+ return false;
+ }
+ return broker.equals(lookupServiceAddress);
+ }
+
+ private NamespaceBundle getNamespaceBundle(String bundle) {
+ final String namespaceName = LoadManagerShared.getNamespaceNameFromBundleName(bundle);
+ final String bundleRange = LoadManagerShared.getBundleRangeFromBundleName(bundle);
+ return pulsar.getNamespaceService().getNamespaceBundleFactory().getBundle(namespaceName, bundleRange);
+ }
+
+ private CompletableFuture<String> deferGetOwnerRequest(String serviceUnit) {
+ return getOwnerRequests
+ .computeIfAbsent(serviceUnit, k -> {
+ CompletableFuture future = new CompletableFuture<>();
+ future.orTimeout(inFlightStateWaitingTimeInMillis, TimeUnit.MILLISECONDS)
+ .whenComplete((v, e) -> {
+ if (e != null) {
+ getOwnerRequests.remove(serviceUnit);
+ log.warn("Failed to getOwner for serviceUnit:{}",
+ serviceUnit, e);
+ future.completeExceptionally((Throwable) e);
+ }
+ }
+ );
+ return future;
+ });
+ }
+
+ private CompletableFuture<Integer> closeServiceUnit(String serviceUnit) {
+ long startTime = System.nanoTime();
+ MutableInt unloadedTopics = new MutableInt();
+ NamespaceBundle bundle = getNamespaceBundle(serviceUnit);
+ return pulsar.getBrokerService().unloadServiceUnit(
+ bundle,
+ false,
+ pulsar.getConfig().getNamespaceBundleUnloadingTimeoutMs(),
+ TimeUnit.MILLISECONDS)
+ .handle((numUnloadedTopics, ex) -> {
+ if (ex != null) {
+ // ignore topic-close failure to unload bundle
+ log.error("Failed to close topics under bundle:{}",
+ bundle.toString(), ex);
+ } else {
+ unloadedTopics.setValue(numUnloadedTopics);
+ }
+ // clean up topics that failed to unload from the broker ownership cache
+ pulsar.getBrokerService().cleanUnloadedTopicFromCache(bundle);
+ return numUnloadedTopics;
+ })
+ .whenComplete((ignored, ex) -> {
+ double unloadBundleTime = TimeUnit.NANOSECONDS
+ .toMillis((System.nanoTime() - startTime));
+ log.info("Unloading bundle:{} with {} topics completed in {} ms",
+ bundle, unloadedTopics, unloadBundleTime, ex);
+ });
+ }
+
+ private CompletableFuture<Void> splitServiceUnit(String serviceUnit) {
+ // TODO: after the split we need to write the child ownerships to BSC instead of ZK.
+ // Use the configured NamespaceBundleSplitAlgorithm
+ long startTime = System.nanoTime();
+ return pulsar.getNamespaceService()
+ .splitAndOwnBundle(getNamespaceBundle(serviceUnit),
+ false,
+ NamespaceBundleSplitAlgorithm.RANGE_EQUALLY_DIVIDE_ALGO,
+ null)
+ .whenComplete((__, ex) -> {
+ double splitBundleTime = TimeUnit.NANOSECONDS
+ .toMillis((System.nanoTime() - startTime));
+ log.info("Splitting {} namespace-bundle completed in {} ms",
Review Comment:
updated.
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,779 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ private static final Schema<ServiceUnitStateData> SCHEMA = Schema.JSON(ServiceUnitStateData.class);
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private String lookupServiceAddress;
+ private LeaderElectionService leaderElectionService;
+ // TODO: define BrokerRegistry
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private Semaphore outstandingPubMessages;
+
+ private Semaphore outstandingCleanupTombstoneMessages;
+ private ScheduledFuture<?> cleanupTasks;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+
+ private ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ ServiceConfiguration conf = pulsar.getConfiguration();
+ this.lookupServiceAddress = pulsar.getAdvertisedAddress() + ":"
+ + (conf.getWebServicePort().isPresent() ? conf.getWebServicePort().get()
+ : conf.getWebServicePortTls().get());
+ this.outstandingPubMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder()
+ .build();
+ this.cleanupJobs =
+ ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder()
+ .build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ }
+
+ public void start() throws PulsarServerException {
+
+ try {
+ if (leaderElectionService != null) {
+ leaderElectionService.close();
+ log.debug("Closed the channel leader election service.");
+ }
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} was elected as the leader."
+ + "Current channel leader is {}",
+ lookupServiceAddress,
+ leaderElectionService.getCurrentLeader());
+ } else {
+ if (leaderElectionService != null) {
+ log.debug("This broker:{} is a follower. "
+ + "Current channel leader is {}",
+ lookupServiceAddress,
+ leaderElectionService.getCurrentLeader());
+ }
+ }
+ });
+ leaderElectionService.start();
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(SCHEMA)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(SCHEMA)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ // TODO: schedule cleanupTasks by brokerRegistry
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+
+ public void close() throws PulsarServerException {
+ try {
+ if (leaderElectionService != null) {
+ leaderElectionService.close();
+ leaderElectionService = null;
+ log.debug("Successfully closed the channel leader election service.");
+ }
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public void scheduleCompaction() throws PulsarServerException {
+ try {
+ Long threshold = pulsar.getAdminClient().topicPolicies()
+ .getCompactionThreshold(TOPIC);
+ if (threshold == null || threshold == 0) {
+ pulsar.getAdminClient().topicPolicies()
+ .setCompactionThreshold(TOPIC, COMPACTION_THRESHOLD);
+ log.info("Scheduled compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ } else {
+ log.info("Already set compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ }
+ } catch (PulsarAdminException e) {
+ throw new PulsarServerException("Failed to schedule compaction.", e);
+ }
+ }
+
+ public String getChannelOwner() {
+ CompletableFuture<Optional<LeaderBroker>> future = leaderElectionService.readCurrentLeader();
Review Comment:
Made it final.
Updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] shibd commented on a diff in pull request #18489: [improve][broker] PIP-192 Added ServiceUnitStateChannelImpl
Posted by GitBox <gi...@apache.org>.
shibd commented on code in PR #18489:
URL: https://github.com/apache/pulsar/pull/18489#discussion_r1027177087
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,761 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ private static final Schema<ServiceUnitStateData> SCHEMA = Schema.JSON(ServiceUnitStateData.class);
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private final String lookupServiceAddress;
+ // TODO: define BrokerRegistry
+ private final Semaphore outstandingCleanupTombstoneMessages;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+ private LeaderElectionService leaderElectionService;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private ScheduledFuture<?> cleanupTasks;
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+ private volatile boolean isActive;
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.isActive = false;
+ this.pulsar = pulsar;
+ ServiceConfiguration conf = pulsar.getConfiguration();
+ this.lookupServiceAddress = pulsar.getLookupServiceAddress();
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder().build();
+ this.cleanupJobs = ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder().build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ }
+
+ public synchronized void start() throws PulsarServerException {
+ try {
+ if (leaderElectionService != null) {
+ leaderElectionService.close();
+ log.debug("Closed the channel leader election service.");
+ }
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} is the leader now.", lookupServiceAddress);
+ // TODO: schedule monitorOwnerships by brokerRegistry
+ } else {
+ log.debug("This broker:{} is a follower now.", lookupServiceAddress);
+ // TODO: cancel scheduled monitorOwnerships if any
+ }
+ });
+ leaderElectionService.start();
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(SCHEMA)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(SCHEMA)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ isActive = true;
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public synchronized void close() throws PulsarServerException {
+ isActive = false;
+ try {
+ if (leaderElectionService != null) {
+ leaderElectionService.close();
+ leaderElectionService = null;
+ log.debug("Successfully closed the channel leader election service.");
+ }
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ private void validateChannel() {
+ if (!isActive) {
+ throw new IllegalStateException("The channel has not been started.");
+ }
+ }
+
+ public synchronized void scheduleCompaction() throws PulsarServerException {
+ try {
+ Long threshold = pulsar.getAdminClient().topicPolicies()
+ .getCompactionThreshold(TOPIC);
+ if (threshold == null || threshold == 0) {
+ pulsar.getAdminClient().topicPolicies()
+ .setCompactionThreshold(TOPIC, COMPACTION_THRESHOLD);
+ log.info("Scheduled compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ } else {
+ log.info("Already set compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ }
+ } catch (PulsarAdminException e) {
+ throw new PulsarServerException("Failed to schedule compaction.", e);
+ }
+ }
+
+ public String getChannelOwner() {
+ validateChannel();
+ CompletableFuture<Optional<LeaderBroker>> future = leaderElectionService.readCurrentLeader();
+ if (!future.isDone()) {
+ return null;
+ }
+ Optional<LeaderBroker> leader = future.join();
+ if (leader.isEmpty()) {
+ return null;
+ }
+ //expecting http://broker-xyz:port
+ // TODO: discard this protocol prefix removal by a util func that returns lookupServiceAddress(serviceUrl)
+ String broker = leader.get().getServiceUrl();
+ broker = broker.substring(broker.lastIndexOf('/') + 1);
+ return broker;
+ }
+
+ public boolean isChannelOwner() {
+ return isTargetBroker(getChannelOwner());
+ }
+
+ public CompletableFuture<String> getOwnerAsync(String serviceUnit) {
+ validateChannel();
+ ServiceUnitStateData data = tableview.get(serviceUnit);
+ if (data == null) {
+ return null;
+ }
+ switch (data.state()) {
+ case Owned, Splitting -> {
+ return CompletableFuture.completedFuture(data.broker());
+ }
+ case Assigned, Released -> {
+ return deferGetOwnerRequest(serviceUnit);
+ }
+ default -> {
+ return null;
+ }
+ }
+ }
+
+ public CompletableFuture<String> publishAssignEventAsync(String serviceUnit, String broker) {
+ CompletableFuture<String> getOwnerRequest = deferGetOwnerRequest(serviceUnit);
+ pubAsync(serviceUnit, new ServiceUnitStateData(Assigned, broker))
+ .whenComplete((__, ex) -> {
+ if (ex != null) {
+ getOwnerRequests.remove(serviceUnit);
+ if (!getOwnerRequest.isCompletedExceptionally()) {
+ getOwnerRequest.completeExceptionally(ex);
+ }
+ }
+ });
+
+ return getOwnerRequest;
+ }
+
+ public CompletableFuture<Void> publishUnloadEventAsync(Unload unload) {
+ String serviceUnit = unload.serviceUnit();
+ if (isTransferCommand(unload)) {
+ ServiceUnitStateData next = new ServiceUnitStateData(Assigned,
Review Comment:
Here should publish `Free` status?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on a diff in pull request #18489: [improve][broker] PIP-192 Added ServiceUnitStateChannelImpl
Posted by GitBox <gi...@apache.org>.
eolivelli commented on code in PR #18489:
URL: https://github.com/apache/pulsar/pull/18489#discussion_r1027253227
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,779 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ private static final Schema<ServiceUnitStateData> SCHEMA = Schema.JSON(ServiceUnitStateData.class);
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private String lookupServiceAddress;
+ private LeaderElectionService leaderElectionService;
+ // TODO: define BrokerRegistry
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private Semaphore outstandingPubMessages;
+
+ private Semaphore outstandingCleanupTombstoneMessages;
+ private ScheduledFuture<?> cleanupTasks;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+
+ private ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ ServiceConfiguration conf = pulsar.getConfiguration();
+ this.lookupServiceAddress = pulsar.getAdvertisedAddress() + ":"
+ + (conf.getWebServicePort().isPresent() ? conf.getWebServicePort().get()
+ : conf.getWebServicePortTls().get());
+ this.outstandingPubMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder()
+ .build();
+ this.cleanupJobs =
+ ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder()
+ .build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ }
+
+ public void start() throws PulsarServerException {
+
+ try {
+ if (leaderElectionService != null) {
+ leaderElectionService.close();
+ log.debug("Closed the channel leader election service.");
+ }
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} was elected as the leader."
+ + "Current channel leader is {}",
+ lookupServiceAddress,
+ leaderElectionService.getCurrentLeader());
+ } else {
+ if (leaderElectionService != null) {
+ log.debug("This broker:{} is a follower. "
+ + "Current channel leader is {}",
+ lookupServiceAddress,
+ leaderElectionService.getCurrentLeader());
+ }
+ }
+ });
+ leaderElectionService.start();
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(SCHEMA)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(SCHEMA)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ // TODO: schedule cleanupTasks by brokerRegistry
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+
+ public void close() throws PulsarServerException {
+ try {
+ if (leaderElectionService != null) {
+ leaderElectionService.close();
+ leaderElectionService = null;
+ log.debug("Successfully closed the channel leader election service.");
+ }
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public void scheduleCompaction() throws PulsarServerException {
+ try {
+ Long threshold = pulsar.getAdminClient().topicPolicies()
+ .getCompactionThreshold(TOPIC);
+ if (threshold == null || threshold == 0) {
+ pulsar.getAdminClient().topicPolicies()
+ .setCompactionThreshold(TOPIC, COMPACTION_THRESHOLD);
+ log.info("Scheduled compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ } else {
+ log.info("Already set compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ }
+ } catch (PulsarAdminException e) {
+ throw new PulsarServerException("Failed to schedule compaction.", e);
+ }
+ }
+
+ public String getChannelOwner() {
+ CompletableFuture<Optional<LeaderBroker>> future = leaderElectionService.readCurrentLeader();
Review Comment:
You are correct we have to handle correctly synchronization around tableview and producer.
The reason is related to the Java Memory Model.
If you access a field from multiple threads you are not guaranteed that each threads see the same value if you use any synchronization mechanism (there are many)
Spot bugs usually finds this problems
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] heesung-sn commented on a diff in pull request #18489: [improve][broker] PIP-192 Added ServiceUnitStateChannelImpl
Posted by GitBox <gi...@apache.org>.
heesung-sn commented on code in PR #18489:
URL: https://github.com/apache/pulsar/pull/18489#discussion_r1034434477
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,778 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Free;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Closed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Constructed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.LeaderElectionServiceStarted;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Started;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
Review Comment:
updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on a diff in pull request #18489: [improve][broker] PIP-192 Added ServiceUnitStateChannelImpl
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on code in PR #18489:
URL: https://github.com/apache/pulsar/pull/18489#discussion_r1035437884
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,778 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Free;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Closed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Constructed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.LeaderElectionServiceStarted;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Started;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private final Schema<ServiceUnitStateData> schema;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private final String lookupServiceAddress;
+ // TODO: define BrokerRegistry
+ private final Semaphore outstandingCleanupTombstoneMessages;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+ private final LeaderElectionService leaderElectionService;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private ScheduledFuture<?> cleanupTasks;
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+ private volatile ChannelState channelState;
+
+ enum ChannelState {
+ Closed(0),
+ Constructed(1),
+ LeaderElectionServiceStarted(2),
+ Started(3);
+
+ ChannelState(int id) {
+ this.id = id;
+ }
+ int id;
+ }
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ this.lookupServiceAddress = pulsar.getLookupServiceAddress();
+ this.schema = Schema.JSON(ServiceUnitStateData.class);
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder().build();
+ this.cleanupJobs = ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder().build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} is the leader now.", lookupServiceAddress);
+ // TODO: schedule monitorOwnerships by brokerRegistry
+ } else {
+ log.debug("This broker:{} is a follower now.", lookupServiceAddress);
+ // TODO: cancel scheduled monitorOwnerships if any
+ }
+ });
+ this.channelState = Constructed;
+ }
+
+ public synchronized void start() throws PulsarServerException {
+ validateChannelState(LeaderElectionServiceStarted, false);
+ try {
+ leaderElectionService.start();
+ this.channelState = LeaderElectionServiceStarted;
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(schema)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(schema)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ channelState = Started;
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public synchronized void close() throws PulsarServerException {
+ channelState = Closed;
+ try {
+ leaderElectionService.close();
+ log.debug("Successfully closed the channel leader election service.");
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ private void validateChannelState(ChannelState targetState, boolean checkLowerIds) {
+ int order = checkLowerIds ? -1 : 1;
+ if (Integer.compare(channelState.id, targetState.id) * order > 0) {
+ throw new IllegalStateException("Invalid channel state:" + channelState.name());
+ }
+ }
+
+ public synchronized void scheduleCompaction() throws PulsarServerException {
+ try {
+ Long threshold = pulsar.getAdminClient().topicPolicies()
+ .getCompactionThreshold(TOPIC);
+ if (threshold == null || threshold == 0) {
+ pulsar.getAdminClient().topicPolicies()
+ .setCompactionThreshold(TOPIC, COMPACTION_THRESHOLD);
+ log.info("Scheduled compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ } else {
+ log.info("Already set compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ }
+ } catch (PulsarAdminException e) {
+ throw new PulsarServerException("Failed to schedule compaction.", e);
+ }
+ }
+
+ public String getChannelOwner() {
+ validateChannelState(LeaderElectionServiceStarted, true);
+ CompletableFuture<Optional<LeaderBroker>> future = leaderElectionService.readCurrentLeader();
+ if (!future.isDone()) {
+ return null;
+ }
+ Optional<LeaderBroker> leader = future.join();
+ if (leader.isEmpty()) {
+ return null;
+ }
+ //expecting http://broker-xyz:port
+ // TODO: discard this protocol prefix removal by a util func that returns lookupServiceAddress(serviceUrl)
+ String broker = leader.get().getServiceUrl();
+ broker = broker.substring(broker.lastIndexOf('/') + 1);
+ return broker;
+ }
+
+ public boolean isChannelOwner() {
+ return isTargetBroker(getChannelOwner());
+ }
+
+ public CompletableFuture<String> getOwnerAsync(String serviceUnit) {
+ validateChannelState(Started, true);
+ ServiceUnitStateData data = tableview.get(serviceUnit);
+ if (data == null) {
+ return null;
+ }
+ switch (data.state()) {
+ case Owned, Splitting -> {
+ return CompletableFuture.completedFuture(data.broker());
+ }
+ case Assigned, Released -> {
+ return deferGetOwnerRequest(serviceUnit);
+ }
+ default -> {
+ return null;
+ }
+ }
+ }
+
+ public CompletableFuture<String> publishAssignEventAsync(String serviceUnit, String broker) {
+ CompletableFuture<String> getOwnerRequest = deferGetOwnerRequest(serviceUnit);
+ pubAsync(serviceUnit, new ServiceUnitStateData(Assigned, broker))
+ .whenComplete((__, ex) -> {
+ if (ex != null) {
+ getOwnerRequests.remove(serviceUnit);
+ if (!getOwnerRequest.isCompletedExceptionally()) {
+ getOwnerRequest.completeExceptionally(ex);
+ }
+ }
+ });
+
+ return getOwnerRequest;
+ }
+
+ public CompletableFuture<Void> publishUnloadEventAsync(Unload unload) {
+ String serviceUnit = unload.serviceUnit();
+ if (isTransferCommand(unload)) {
+ ServiceUnitStateData next = new ServiceUnitStateData(Assigned,
+ unload.destBroker().get(), unload.sourceBroker());
+ return pubAsync(serviceUnit, next).thenAccept(__ -> {
+ });
+ }
+ return tombstoneAsync(serviceUnit).thenAccept(__ -> {
+ });
+ }
+
+ public CompletableFuture<Void> publishSplitEventAsync(Split split) {
+ String serviceUnit = split.serviceUnit();
+ ServiceUnitStateData next = new ServiceUnitStateData(Splitting, split.sourceBroker());
+ return pubAsync(serviceUnit, next).thenAccept(__ -> {
+ });
+ }
+
+ private void handle(String serviceUnit, ServiceUnitStateData data) {
+ if (log.isDebugEnabled()) {
+ log.info("{} received a handle request for serviceUnit:{}, data:{}",
+ lookupServiceAddress, serviceUnit, data);
+ }
+
+ ServiceUnitState state = data == null ? Free : data.state();
+
+ // TODO : Add state validation in tableview by the compaction strategy
+ switch (state) {
+ case Owned -> handleOwnEvent(serviceUnit, data);
+ case Assigned -> handleAssignEvent(serviceUnit, data);
+ case Released -> handleReleaseEvent(serviceUnit, data);
+ case Splitting -> handleSplitEvent(serviceUnit, data);
+ case Free -> handleFreeEvent(serviceUnit);
+ default -> throw new IllegalStateException("Failed to handle channel data:" + data);
+ }
+ }
+
+ private static boolean isTransferCommand(ServiceUnitStateData data) {
+ if (data == null) {
+ return false;
+ }
+ return StringUtils.isNotEmpty(data.sourceBroker());
+ }
+
+ private static boolean isTransferCommand(Unload data) {
+ return data.destBroker().isPresent();
+ }
+
+ private static String getLogEventTag(ServiceUnitStateData data) {
+ return data == null ? "Free" :
+ isTransferCommand(data) ? "Transfer:" + data.state() : data.state().toString();
+ }
+
+ private void log(Throwable e, String serviceUnit, ServiceUnitStateData data, ServiceUnitStateData next) {
+ if (e == null) {
+ if (log.isDebugEnabled() || isTransferCommand(data)) {
+ log.info("{} handled {} event for serviceUnit:{}, cur:{}, next:{}",
+ lookupServiceAddress, getLogEventTag(data), serviceUnit,
+ data == null ? "" : data,
+ next == null ? "" : next);
+ }
+ } else {
+ log.error("{} failed to handle {} event for serviceUnit:{}, cur:{}, next:{}",
+ lookupServiceAddress, getLogEventTag(data), serviceUnit,
+ data == null ? "" : data,
+ next == null ? "" : next,
+ e);
+ }
+ }
+
+ private void handleOwnEvent(String serviceUnit, ServiceUnitStateData data) {
+ var getOwnerRequest = getOwnerRequests.remove(serviceUnit);
+ if (getOwnerRequest != null) {
+ getOwnerRequest.complete(data.broker());
+ }
+ if (isTargetBroker(data.broker())) {
+ log(null, serviceUnit, data, null);
+ }
+
+ }
+
+ private void handleAssignEvent(String serviceUnit, ServiceUnitStateData data) {
+ deferGetOwnerRequest(serviceUnit);
+ if (isTargetBroker(data.broker())) {
+ ServiceUnitStateData next = new ServiceUnitStateData(
+ isTransferCommand(data) ? Released : Owned, data.broker(), data.sourceBroker());
+ pubAsync(serviceUnit, next)
+ .whenComplete((__, e) -> log(e, serviceUnit, data, next));
+ }
+ }
+
+ private void handleReleaseEvent(String serviceUnit, ServiceUnitStateData data) {
+
+ if (isTargetBroker(data.sourceBroker())) {
+ ServiceUnitStateData next = new ServiceUnitStateData(Owned, data.broker(), data.sourceBroker());
+ // TODO: when close, pass message to clients to connect to the new broker
+ closeServiceUnit(serviceUnit)
+ .thenCompose(__ -> pubAsync(serviceUnit, next))
+ .whenComplete((__, e) -> log(e, serviceUnit, data, next));
+ }
+ }
+
+ private void handleSplitEvent(String serviceUnit, ServiceUnitStateData data) {
+ if (isTargetBroker(data.broker())) {
+ splitServiceUnit(serviceUnit)
+ .thenCompose(__ -> tombstoneAsync(serviceUnit))
+ .whenComplete((__, e) -> log(e, serviceUnit, data, null));
+ }
+ }
+
+ private void handleFreeEvent(String serviceUnit) {
+ closeServiceUnit(serviceUnit)
+ .thenAccept(__ -> {
+ var request = getOwnerRequests.remove(serviceUnit);
+ if (request != null) {
+ request.completeExceptionally(new IllegalStateException("The ownership has been unloaded. "
+ + "No owner is found for serviceUnit: " + serviceUnit));
+ }
+ })
+ .whenComplete((__, e) -> log(e, serviceUnit, null, null));
+ }
+
+ private CompletableFuture<MessageId> pubAsync(String serviceUnit, ServiceUnitStateData data) {
+ validateChannelState(Started, true);
+ CompletableFuture<MessageId> future = new CompletableFuture<>();
+ producer.newMessage()
+ .key(serviceUnit)
+ .value(data)
+ .sendAsync()
+ .whenComplete((messageId, e) -> {
+ if (e != null) {
+ log.error("Failed to publish the message: serviceUnit:{}, data:{}",
+ serviceUnit, data, e);
+ future.completeExceptionally(e);
+ } else {
+ future.complete(messageId);
+ }
+ });
+ return future;
+ }
+
+ private CompletableFuture<MessageId> tombstoneAsync(String serviceUnit) {
+ return pubAsync(serviceUnit, null);
+ }
+
+ private boolean isTargetBroker(String broker) {
+ if (broker == null) {
+ return false;
+ }
+ return broker.equals(lookupServiceAddress);
+ }
+
+ private NamespaceBundle getNamespaceBundle(String bundle) {
+ final String namespaceName = LoadManagerShared.getNamespaceNameFromBundleName(bundle);
+ final String bundleRange = LoadManagerShared.getBundleRangeFromBundleName(bundle);
+ return pulsar.getNamespaceService().getNamespaceBundleFactory().getBundle(namespaceName, bundleRange);
+ }
+
+ private CompletableFuture<String> deferGetOwnerRequest(String serviceUnit) {
+ return getOwnerRequests
+ .computeIfAbsent(serviceUnit, k -> {
+ CompletableFuture future = new CompletableFuture<>();
+ future.orTimeout(inFlightStateWaitingTimeInMillis, TimeUnit.MILLISECONDS)
+ .whenComplete((v, e) -> {
+ if (e != null) {
+ getOwnerRequests.remove(serviceUnit);
+ log.warn("Failed to getOwner for serviceUnit:{}",
+ serviceUnit, e);
+ future.completeExceptionally((Throwable) e);
Review Comment:
@heesung-sn I want to say the future should be completed with the exception. So we don't need to call `future.completeExceptionally((Throwable) e);` again?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] heesung-sn commented on a diff in pull request #18489: [improve][broker] PIP-192 Added ServiceUnitStateChannelImpl
Posted by GitBox <gi...@apache.org>.
heesung-sn commented on code in PR #18489:
URL: https://github.com/apache/pulsar/pull/18489#discussion_r1027259946
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,779 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ private static final Schema<ServiceUnitStateData> SCHEMA = Schema.JSON(ServiceUnitStateData.class);
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private String lookupServiceAddress;
+ private LeaderElectionService leaderElectionService;
+ // TODO: define BrokerRegistry
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private Semaphore outstandingPubMessages;
+
+ private Semaphore outstandingCleanupTombstoneMessages;
+ private ScheduledFuture<?> cleanupTasks;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+
+ private ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ ServiceConfiguration conf = pulsar.getConfiguration();
+ this.lookupServiceAddress = pulsar.getAdvertisedAddress() + ":"
+ + (conf.getWebServicePort().isPresent() ? conf.getWebServicePort().get()
+ : conf.getWebServicePortTls().get());
+ this.outstandingPubMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder()
+ .build();
+ this.cleanupJobs =
+ ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder()
+ .build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ }
+
+ public void start() throws PulsarServerException {
+
+ try {
+ if (leaderElectionService != null) {
+ leaderElectionService.close();
+ log.debug("Closed the channel leader election service.");
+ }
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} was elected as the leader."
+ + "Current channel leader is {}",
+ lookupServiceAddress,
+ leaderElectionService.getCurrentLeader());
+ } else {
+ if (leaderElectionService != null) {
+ log.debug("This broker:{} is a follower. "
+ + "Current channel leader is {}",
+ lookupServiceAddress,
+ leaderElectionService.getCurrentLeader());
+ }
+ }
+ });
+ leaderElectionService.start();
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(SCHEMA)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(SCHEMA)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ // TODO: schedule cleanupTasks by brokerRegistry
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+
+ public void close() throws PulsarServerException {
+ try {
+ if (leaderElectionService != null) {
+ leaderElectionService.close();
+ leaderElectionService = null;
+ log.debug("Successfully closed the channel leader election service.");
+ }
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public void scheduleCompaction() throws PulsarServerException {
+ try {
+ Long threshold = pulsar.getAdminClient().topicPolicies()
+ .getCompactionThreshold(TOPIC);
+ if (threshold == null || threshold == 0) {
+ pulsar.getAdminClient().topicPolicies()
+ .setCompactionThreshold(TOPIC, COMPACTION_THRESHOLD);
+ log.info("Scheduled compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ } else {
+ log.info("Already set compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ }
+ } catch (PulsarAdminException e) {
+ throw new PulsarServerException("Failed to schedule compaction.", e);
+ }
+ }
+
+ public String getChannelOwner() {
+ CompletableFuture<Optional<LeaderBroker>> future = leaderElectionService.readCurrentLeader();
Review Comment:
Aren't the current state channel validation and synchronized start/close func enough? The invalid channel state can throw exception when the getter funcs are called during start() and close().
If this is not sufficient, can you explain why? And do you suggest forcefully synchronized access like 'synchronized getProducer()/setProducer() and synchronized getTableView()/setTableView()?
We could use ReadWriteLock to optimize this, but isn't it an overkill?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] heesung-sn commented on a diff in pull request #18489: [improve][broker] PIP-192 Added ServiceUnitStateChannelImpl
Posted by GitBox <gi...@apache.org>.
heesung-sn commented on code in PR #18489:
URL: https://github.com/apache/pulsar/pull/18489#discussion_r1027241928
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,779 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ private static final Schema<ServiceUnitStateData> SCHEMA = Schema.JSON(ServiceUnitStateData.class);
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private String lookupServiceAddress;
+ private LeaderElectionService leaderElectionService;
+ // TODO: define BrokerRegistry
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private Semaphore outstandingPubMessages;
+
+ private Semaphore outstandingCleanupTombstoneMessages;
+ private ScheduledFuture<?> cleanupTasks;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+
+ private ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ ServiceConfiguration conf = pulsar.getConfiguration();
+ this.lookupServiceAddress = pulsar.getAdvertisedAddress() + ":"
+ + (conf.getWebServicePort().isPresent() ? conf.getWebServicePort().get()
+ : conf.getWebServicePortTls().get());
+ this.outstandingPubMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder()
+ .build();
+ this.cleanupJobs =
+ ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder()
+ .build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ }
+
+ public void start() throws PulsarServerException {
+
+ try {
+ if (leaderElectionService != null) {
+ leaderElectionService.close();
+ log.debug("Closed the channel leader election service.");
+ }
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} was elected as the leader."
+ + "Current channel leader is {}",
+ lookupServiceAddress,
+ leaderElectionService.getCurrentLeader());
+ } else {
+ if (leaderElectionService != null) {
+ log.debug("This broker:{} is a follower. "
+ + "Current channel leader is {}",
+ lookupServiceAddress,
+ leaderElectionService.getCurrentLeader());
+ }
+ }
+ });
+ leaderElectionService.start();
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(SCHEMA)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(SCHEMA)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ // TODO: schedule cleanupTasks by brokerRegistry
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+
+ public void close() throws PulsarServerException {
+ try {
+ if (leaderElectionService != null) {
+ leaderElectionService.close();
+ leaderElectionService = null;
+ log.debug("Successfully closed the channel leader election service.");
+ }
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public void scheduleCompaction() throws PulsarServerException {
+ try {
+ Long threshold = pulsar.getAdminClient().topicPolicies()
+ .getCompactionThreshold(TOPIC);
+ if (threshold == null || threshold == 0) {
+ pulsar.getAdminClient().topicPolicies()
+ .setCompactionThreshold(TOPIC, COMPACTION_THRESHOLD);
+ log.info("Scheduled compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ } else {
+ log.info("Already set compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ }
+ } catch (PulsarAdminException e) {
+ throw new PulsarServerException("Failed to schedule compaction.", e);
+ }
+ }
+
+ public String getChannelOwner() {
+ CompletableFuture<Optional<LeaderBroker>> future = leaderElectionService.readCurrentLeader();
Review Comment:
About the concurrent access to the member variables, still 'tableview' and 'producer' are not final.
Could you explain more why we need the synchronized access to the 'leaderElectionService' instance?
As long as it's non-null, shouldn't it be ok whether the return(leader) is from the old or new instance of 'leaderElectionService'?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] heesung-sn commented on a diff in pull request #18489: [improve][broker] PIP-192 Added ServiceUnitStateChannelImpl
Posted by GitBox <gi...@apache.org>.
heesung-sn commented on code in PR #18489:
URL: https://github.com/apache/pulsar/pull/18489#discussion_r1026849908
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,779 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ private static final Schema<ServiceUnitStateData> SCHEMA = Schema.JSON(ServiceUnitStateData.class);
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private String lookupServiceAddress;
+ private LeaderElectionService leaderElectionService;
+ // TODO: define BrokerRegistry
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private Semaphore outstandingPubMessages;
+
+ private Semaphore outstandingCleanupTombstoneMessages;
+ private ScheduledFuture<?> cleanupTasks;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+
+ private ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ ServiceConfiguration conf = pulsar.getConfiguration();
+ this.lookupServiceAddress = pulsar.getAdvertisedAddress() + ":"
+ + (conf.getWebServicePort().isPresent() ? conf.getWebServicePort().get()
+ : conf.getWebServicePortTls().get());
+ this.outstandingPubMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder()
+ .build();
+ this.cleanupJobs =
+ ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder()
+ .build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ }
+
+ public void start() throws PulsarServerException {
+
+ try {
+ if (leaderElectionService != null) {
+ leaderElectionService.close();
+ log.debug("Closed the channel leader election service.");
+ }
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} was elected as the leader."
+ + "Current channel leader is {}",
+ lookupServiceAddress,
+ leaderElectionService.getCurrentLeader());
+ } else {
+ if (leaderElectionService != null) {
+ log.debug("This broker:{} is a follower. "
+ + "Current channel leader is {}",
+ lookupServiceAddress,
+ leaderElectionService.getCurrentLeader());
+ }
+ }
+ });
+ leaderElectionService.start();
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(SCHEMA)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(SCHEMA)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ // TODO: schedule cleanupTasks by brokerRegistry
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+
+ public void close() throws PulsarServerException {
+ try {
+ if (leaderElectionService != null) {
+ leaderElectionService.close();
+ leaderElectionService = null;
+ log.debug("Successfully closed the channel leader election service.");
+ }
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public void scheduleCompaction() throws PulsarServerException {
+ try {
+ Long threshold = pulsar.getAdminClient().topicPolicies()
+ .getCompactionThreshold(TOPIC);
+ if (threshold == null || threshold == 0) {
+ pulsar.getAdminClient().topicPolicies()
+ .setCompactionThreshold(TOPIC, COMPACTION_THRESHOLD);
+ log.info("Scheduled compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ } else {
+ log.info("Already set compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ }
+ } catch (PulsarAdminException e) {
+ throw new PulsarServerException("Failed to schedule compaction.", e);
+ }
+ }
+
+ public String getChannelOwner() {
+ CompletableFuture<Optional<LeaderBroker>> future = leaderElectionService.readCurrentLeader();
+ if (!future.isDone()) {
+ return null;
+ }
+ Optional<LeaderBroker> leader = future.join();
+ if (leader.isEmpty()) {
+ return null;
+ }
+ //expecting http://broker-xyz:port
+ String broker = leader.get().getServiceUrl();
+ broker = broker.substring(broker.lastIndexOf('/') + 1);
+ return broker;
+ }
+
+ public boolean isChannelOwner() {
+ return isTargetBroker(getChannelOwner());
+ }
+
+ public CompletableFuture<String> getOwnerAsync(String serviceUnit) {
+
+ ServiceUnitStateData data = tableview.get(serviceUnit);
+ if (data == null) {
+ return null;
+ }
+ switch (data.state()) {
+ case Owned, Splitting -> {
+ return CompletableFuture.completedFuture(data.broker());
+ }
+ case Assigned, Released -> {
+ return deferGetOwnerRequest(serviceUnit);
+ }
+ default -> {
+ return null;
+ }
+ }
+ }
+
+ public CompletableFuture<String> publishAssignEventAsync(String serviceUnit, String broker) {
+ CompletableFuture<String> getOwnerRequest = deferGetOwnerRequest(serviceUnit);
+ pubAsync(serviceUnit, new ServiceUnitStateData(Assigned, broker))
+ .whenComplete((__, ex) -> {
+ if (ex != null) {
+ getOwnerRequests.remove(serviceUnit);
+ if (!getOwnerRequest.isCompletedExceptionally()) {
+ getOwnerRequest.completeExceptionally(ex);
+ }
+ }
+ });
+
+ return getOwnerRequest;
+ }
+
+ public CompletableFuture<Void> publishUnloadEventAsync(Unload unload) {
+ String serviceUnit = unload.serviceUnit();
+ if (isTransferCommand(unload)) {
+ ServiceUnitStateData next = new ServiceUnitStateData(Assigned,
+ unload.destBroker().get(), unload.sourceBroker());
+ return pubAsync(serviceUnit, next).thenAccept(__ -> {
+ });
+ }
+ return tombstoneAsync(serviceUnit).thenAccept(__ -> {
+ });
+ }
+
+ public CompletableFuture<Void> publishSplitEventAsync(Split split) {
+ String serviceUnit = split.serviceUnit();
+ ServiceUnitStateData data = tableview.get(serviceUnit);
+ ServiceUnitStateData next = new ServiceUnitStateData(Splitting, data.broker());
+ return pubAsync(serviceUnit, next).thenAccept(__ -> {
+ });
+ }
+
+ private void handle(String serviceUnit, ServiceUnitStateData data) {
+ if (log.isDebugEnabled()) {
+ log.info("{} received a handle request for serviceUnit:{}, data:{}",
+ lookupServiceAddress, serviceUnit, data);
+ }
+
+ if (data == null) {
+ handleTombstoneEvent(serviceUnit);
+ return;
+ }
+
+ // TODO : Add state validation
+ switch (data.state()) {
+ case Owned -> handleOwnEvent(serviceUnit, data);
+ case Assigned -> handleAssignEvent(serviceUnit, data);
+ case Released -> handleReleaseEvent(serviceUnit, data);
+ case Splitting -> handleSplitEvent(serviceUnit, data);
+ default -> throw new IllegalStateException("Failed to handle channel data:" + data);
+ }
+ }
+
+ private static boolean isTransferCommand(ServiceUnitStateData data) {
+ if (data == null) {
+ return false;
+ }
+ return StringUtils.isNotEmpty(data.sourceBroker());
+ }
+
+ private static boolean isTransferCommand(Unload data) {
+ return data.destBroker().isPresent();
+ }
+
+ private static String getLogEventTag(ServiceUnitStateData data) {
+ return data == null ? "Free" :
+ isTransferCommand(data) ? "Transfer:" + data.state() : data.state().toString();
+ }
+
+ private void log(Throwable e, String serviceUnit, ServiceUnitStateData data, ServiceUnitStateData next) {
Review Comment:
No. It can't. `lookupServiceAddress` is a member var.
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,779 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ private static final Schema<ServiceUnitStateData> SCHEMA = Schema.JSON(ServiceUnitStateData.class);
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private String lookupServiceAddress;
+ private LeaderElectionService leaderElectionService;
+ // TODO: define BrokerRegistry
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private Semaphore outstandingPubMessages;
+
+ private Semaphore outstandingCleanupTombstoneMessages;
+ private ScheduledFuture<?> cleanupTasks;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+
+ private ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ ServiceConfiguration conf = pulsar.getConfiguration();
+ this.lookupServiceAddress = pulsar.getAdvertisedAddress() + ":"
+ + (conf.getWebServicePort().isPresent() ? conf.getWebServicePort().get()
+ : conf.getWebServicePortTls().get());
+ this.outstandingPubMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder()
+ .build();
+ this.cleanupJobs =
+ ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder()
+ .build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ }
+
+ public void start() throws PulsarServerException {
+
+ try {
+ if (leaderElectionService != null) {
+ leaderElectionService.close();
+ log.debug("Closed the channel leader election service.");
+ }
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} was elected as the leader."
+ + "Current channel leader is {}",
+ lookupServiceAddress,
+ leaderElectionService.getCurrentLeader());
+ } else {
+ if (leaderElectionService != null) {
+ log.debug("This broker:{} is a follower. "
+ + "Current channel leader is {}",
+ lookupServiceAddress,
+ leaderElectionService.getCurrentLeader());
+ }
+ }
+ });
+ leaderElectionService.start();
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(SCHEMA)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(SCHEMA)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ // TODO: schedule cleanupTasks by brokerRegistry
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+
+ public void close() throws PulsarServerException {
+ try {
+ if (leaderElectionService != null) {
+ leaderElectionService.close();
+ leaderElectionService = null;
+ log.debug("Successfully closed the channel leader election service.");
+ }
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public void scheduleCompaction() throws PulsarServerException {
+ try {
+ Long threshold = pulsar.getAdminClient().topicPolicies()
+ .getCompactionThreshold(TOPIC);
+ if (threshold == null || threshold == 0) {
+ pulsar.getAdminClient().topicPolicies()
+ .setCompactionThreshold(TOPIC, COMPACTION_THRESHOLD);
+ log.info("Scheduled compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ } else {
+ log.info("Already set compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ }
+ } catch (PulsarAdminException e) {
+ throw new PulsarServerException("Failed to schedule compaction.", e);
+ }
+ }
+
+ public String getChannelOwner() {
+ CompletableFuture<Optional<LeaderBroker>> future = leaderElectionService.readCurrentLeader();
+ if (!future.isDone()) {
+ return null;
+ }
+ Optional<LeaderBroker> leader = future.join();
+ if (leader.isEmpty()) {
+ return null;
+ }
+ //expecting http://broker-xyz:port
+ String broker = leader.get().getServiceUrl();
+ broker = broker.substring(broker.lastIndexOf('/') + 1);
+ return broker;
+ }
+
+ public boolean isChannelOwner() {
+ return isTargetBroker(getChannelOwner());
+ }
+
+ public CompletableFuture<String> getOwnerAsync(String serviceUnit) {
+
+ ServiceUnitStateData data = tableview.get(serviceUnit);
+ if (data == null) {
+ return null;
+ }
+ switch (data.state()) {
+ case Owned, Splitting -> {
+ return CompletableFuture.completedFuture(data.broker());
+ }
+ case Assigned, Released -> {
+ return deferGetOwnerRequest(serviceUnit);
+ }
+ default -> {
+ return null;
+ }
+ }
+ }
+
+ public CompletableFuture<String> publishAssignEventAsync(String serviceUnit, String broker) {
+ CompletableFuture<String> getOwnerRequest = deferGetOwnerRequest(serviceUnit);
+ pubAsync(serviceUnit, new ServiceUnitStateData(Assigned, broker))
+ .whenComplete((__, ex) -> {
+ if (ex != null) {
+ getOwnerRequests.remove(serviceUnit);
+ if (!getOwnerRequest.isCompletedExceptionally()) {
+ getOwnerRequest.completeExceptionally(ex);
+ }
+ }
+ });
+
+ return getOwnerRequest;
+ }
+
+ public CompletableFuture<Void> publishUnloadEventAsync(Unload unload) {
+ String serviceUnit = unload.serviceUnit();
+ if (isTransferCommand(unload)) {
+ ServiceUnitStateData next = new ServiceUnitStateData(Assigned,
+ unload.destBroker().get(), unload.sourceBroker());
+ return pubAsync(serviceUnit, next).thenAccept(__ -> {
+ });
+ }
+ return tombstoneAsync(serviceUnit).thenAccept(__ -> {
+ });
+ }
+
+ public CompletableFuture<Void> publishSplitEventAsync(Split split) {
+ String serviceUnit = split.serviceUnit();
+ ServiceUnitStateData data = tableview.get(serviceUnit);
+ ServiceUnitStateData next = new ServiceUnitStateData(Splitting, data.broker());
+ return pubAsync(serviceUnit, next).thenAccept(__ -> {
+ });
+ }
+
+ private void handle(String serviceUnit, ServiceUnitStateData data) {
+ if (log.isDebugEnabled()) {
+ log.info("{} received a handle request for serviceUnit:{}, data:{}",
+ lookupServiceAddress, serviceUnit, data);
+ }
+
+ if (data == null) {
+ handleTombstoneEvent(serviceUnit);
+ return;
+ }
+
+ // TODO : Add state validation
+ switch (data.state()) {
+ case Owned -> handleOwnEvent(serviceUnit, data);
+ case Assigned -> handleAssignEvent(serviceUnit, data);
+ case Released -> handleReleaseEvent(serviceUnit, data);
+ case Splitting -> handleSplitEvent(serviceUnit, data);
+ default -> throw new IllegalStateException("Failed to handle channel data:" + data);
+ }
+ }
+
+ private static boolean isTransferCommand(ServiceUnitStateData data) {
+ if (data == null) {
+ return false;
+ }
+ return StringUtils.isNotEmpty(data.sourceBroker());
+ }
+
+ private static boolean isTransferCommand(Unload data) {
+ return data.destBroker().isPresent();
+ }
+
+ private static String getLogEventTag(ServiceUnitStateData data) {
+ return data == null ? "Free" :
+ isTransferCommand(data) ? "Transfer:" + data.state() : data.state().toString();
+ }
+
+ private void log(Throwable e, String serviceUnit, ServiceUnitStateData data, ServiceUnitStateData next) {
+ if (e == null) {
+ if (log.isDebugEnabled() || isTransferCommand(data)) {
+ log.info("{} handled {} event for serviceUnit:{}, cur:{}, next:{}",
+ lookupServiceAddress, getLogEventTag(data), serviceUnit,
+ data == null ? "" : data,
+ next == null ? "" : next);
+ }
+ } else {
+ log.error("{} failed to handle {} event for serviceUnit:{}, cur:{}, next:{}",
+ lookupServiceAddress, getLogEventTag(data), serviceUnit,
+ data == null ? "" : data,
+ next == null ? "" : next,
+ e);
+ }
+ }
+
+ private void handleOwnEvent(String serviceUnit, ServiceUnitStateData data) {
+ var getOwnerRequest = getOwnerRequests.remove(serviceUnit);
+ if (getOwnerRequest != null) {
+ getOwnerRequest.complete(data.broker());
+ }
+ if (isTargetBroker(data.broker())) {
+ log(null, serviceUnit, data, null);
+ }
+
+ }
+
+ private void handleAssignEvent(String serviceUnit, ServiceUnitStateData data) {
+ deferGetOwnerRequest(serviceUnit);
+ if (isTargetBroker(data.broker())) {
+ ServiceUnitStateData next = new ServiceUnitStateData(
+ isTransferCommand(data) ? Released : Owned, data.broker(), data.sourceBroker());
+ pubAsync(serviceUnit, next)
+ .whenComplete((__, e) -> log(e, serviceUnit, data, next));
+ }
+ }
+
+ private void handleReleaseEvent(String serviceUnit, ServiceUnitStateData data) {
+
+ if (isTargetBroker(data.sourceBroker())) {
+ ServiceUnitStateData next = new ServiceUnitStateData(Owned, data.broker(), data.sourceBroker());
+ // TODO: when close, pass message to clients to connect to the new broker
+ closeServiceUnit(serviceUnit)
+ .thenCompose(__ -> pubAsync(serviceUnit, next))
+ .whenComplete((__, e) -> log(e, serviceUnit, data, next));
+ }
+ }
+
+ private void handleSplitEvent(String serviceUnit, ServiceUnitStateData data) {
+ if (isTargetBroker(data.broker())) {
+ splitServiceUnit(serviceUnit)
+ .thenCompose(__ -> tombstoneAsync(serviceUnit))
+ .whenComplete((__, e) -> log(e, serviceUnit, data, null));
+ }
+ }
+
+ private void handleTombstoneEvent(String serviceUnit) {
+ closeServiceUnit(serviceUnit)
+ .thenAccept(__ -> {
+ var request = getOwnerRequests.remove(serviceUnit);
+ if (request != null) {
+ request.completeExceptionally(new IllegalStateException("The ownership has been unloaded. "
+ + "No owner is found for serviceUnit: " + serviceUnit));
+ }
+ })
+ .whenComplete((__, e) -> log(e, serviceUnit, null, null));
+ }
+
+ private CompletableFuture<MessageId> pubAsync(String serviceUnit, ServiceUnitStateData data) {
+ CompletableFuture<MessageId> future = new CompletableFuture<>();
+ try {
+ outstandingPubMessages.acquire();
Review Comment:
The intention was to limit the `pubAsync` concurrency from this channel not to exhaust the broker threads.
I removed this semaphore.
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,779 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ private static final Schema<ServiceUnitStateData> SCHEMA = Schema.JSON(ServiceUnitStateData.class);
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private String lookupServiceAddress;
+ private LeaderElectionService leaderElectionService;
+ // TODO: define BrokerRegistry
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private Semaphore outstandingPubMessages;
+
+ private Semaphore outstandingCleanupTombstoneMessages;
+ private ScheduledFuture<?> cleanupTasks;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+
+ private ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ ServiceConfiguration conf = pulsar.getConfiguration();
+ this.lookupServiceAddress = pulsar.getAdvertisedAddress() + ":"
+ + (conf.getWebServicePort().isPresent() ? conf.getWebServicePort().get()
+ : conf.getWebServicePortTls().get());
+ this.outstandingPubMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder()
+ .build();
+ this.cleanupJobs =
+ ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder()
+ .build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ }
+
+ public void start() throws PulsarServerException {
+
+ try {
+ if (leaderElectionService != null) {
+ leaderElectionService.close();
+ log.debug("Closed the channel leader election service.");
+ }
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} was elected as the leader."
+ + "Current channel leader is {}",
+ lookupServiceAddress,
+ leaderElectionService.getCurrentLeader());
+ } else {
+ if (leaderElectionService != null) {
+ log.debug("This broker:{} is a follower. "
+ + "Current channel leader is {}",
+ lookupServiceAddress,
+ leaderElectionService.getCurrentLeader());
+ }
+ }
+ });
+ leaderElectionService.start();
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(SCHEMA)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(SCHEMA)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ // TODO: schedule cleanupTasks by brokerRegistry
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+
+ public void close() throws PulsarServerException {
+ try {
+ if (leaderElectionService != null) {
+ leaderElectionService.close();
+ leaderElectionService = null;
+ log.debug("Successfully closed the channel leader election service.");
+ }
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public void scheduleCompaction() throws PulsarServerException {
+ try {
+ Long threshold = pulsar.getAdminClient().topicPolicies()
+ .getCompactionThreshold(TOPIC);
+ if (threshold == null || threshold == 0) {
+ pulsar.getAdminClient().topicPolicies()
+ .setCompactionThreshold(TOPIC, COMPACTION_THRESHOLD);
+ log.info("Scheduled compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ } else {
+ log.info("Already set compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ }
+ } catch (PulsarAdminException e) {
+ throw new PulsarServerException("Failed to schedule compaction.", e);
+ }
+ }
+
+ public String getChannelOwner() {
+ CompletableFuture<Optional<LeaderBroker>> future = leaderElectionService.readCurrentLeader();
Review Comment:
- Added a private variable `volatile boolean isActive` and set this flag in start() and close().
- Made start() and close() synchronized.
- Added validateChannel() to validate concurrent access to `leaderElectionService, producer, and tableview` while `isActive=false`.
- Added a UT to cover this access logic.
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,779 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ private static final Schema<ServiceUnitStateData> SCHEMA = Schema.JSON(ServiceUnitStateData.class);
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private String lookupServiceAddress;
+ private LeaderElectionService leaderElectionService;
+ // TODO: define BrokerRegistry
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private Semaphore outstandingPubMessages;
+
+ private Semaphore outstandingCleanupTombstoneMessages;
+ private ScheduledFuture<?> cleanupTasks;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+
+ private ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ ServiceConfiguration conf = pulsar.getConfiguration();
+ this.lookupServiceAddress = pulsar.getAdvertisedAddress() + ":"
+ + (conf.getWebServicePort().isPresent() ? conf.getWebServicePort().get()
+ : conf.getWebServicePortTls().get());
+ this.outstandingPubMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder()
+ .build();
+ this.cleanupJobs =
+ ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder()
+ .build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ }
+
+ public void start() throws PulsarServerException {
+
+ try {
+ if (leaderElectionService != null) {
+ leaderElectionService.close();
+ log.debug("Closed the channel leader election service.");
+ }
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} was elected as the leader."
+ + "Current channel leader is {}",
+ lookupServiceAddress,
+ leaderElectionService.getCurrentLeader());
+ } else {
+ if (leaderElectionService != null) {
+ log.debug("This broker:{} is a follower. "
+ + "Current channel leader is {}",
+ lookupServiceAddress,
+ leaderElectionService.getCurrentLeader());
+ }
+ }
+ });
+ leaderElectionService.start();
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(SCHEMA)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(SCHEMA)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ // TODO: schedule cleanupTasks by brokerRegistry
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+
+ public void close() throws PulsarServerException {
+ try {
+ if (leaderElectionService != null) {
+ leaderElectionService.close();
+ leaderElectionService = null;
+ log.debug("Successfully closed the channel leader election service.");
+ }
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public void scheduleCompaction() throws PulsarServerException {
+ try {
+ Long threshold = pulsar.getAdminClient().topicPolicies()
+ .getCompactionThreshold(TOPIC);
+ if (threshold == null || threshold == 0) {
+ pulsar.getAdminClient().topicPolicies()
+ .setCompactionThreshold(TOPIC, COMPACTION_THRESHOLD);
+ log.info("Scheduled compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ } else {
+ log.info("Already set compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ }
+ } catch (PulsarAdminException e) {
+ throw new PulsarServerException("Failed to schedule compaction.", e);
+ }
+ }
+
+ public String getChannelOwner() {
+ CompletableFuture<Optional<LeaderBroker>> future = leaderElectionService.readCurrentLeader();
+ if (!future.isDone()) {
+ return null;
+ }
+ Optional<LeaderBroker> leader = future.join();
+ if (leader.isEmpty()) {
+ return null;
+ }
+ //expecting http://broker-xyz:port
+ String broker = leader.get().getServiceUrl();
+ broker = broker.substring(broker.lastIndexOf('/') + 1);
Review Comment:
Agreed, but I assume this util func is not required in this PR. I added a TODO comment here.
`// TODO: discard this protocol prefix removal by a util func that returns lookupServiceAddress(serviceUrl)`
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,779 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ private static final Schema<ServiceUnitStateData> SCHEMA = Schema.JSON(ServiceUnitStateData.class);
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private String lookupServiceAddress;
+ private LeaderElectionService leaderElectionService;
+ // TODO: define BrokerRegistry
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private Semaphore outstandingPubMessages;
+
+ private Semaphore outstandingCleanupTombstoneMessages;
+ private ScheduledFuture<?> cleanupTasks;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+
+ private ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ ServiceConfiguration conf = pulsar.getConfiguration();
+ this.lookupServiceAddress = pulsar.getAdvertisedAddress() + ":"
+ + (conf.getWebServicePort().isPresent() ? conf.getWebServicePort().get()
+ : conf.getWebServicePortTls().get());
+ this.outstandingPubMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder()
+ .build();
+ this.cleanupJobs =
+ ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder()
+ .build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ }
+
+ public void start() throws PulsarServerException {
+
+ try {
+ if (leaderElectionService != null) {
+ leaderElectionService.close();
+ log.debug("Closed the channel leader election service.");
+ }
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} was elected as the leader."
+ + "Current channel leader is {}",
+ lookupServiceAddress,
+ leaderElectionService.getCurrentLeader());
+ } else {
+ if (leaderElectionService != null) {
+ log.debug("This broker:{} is a follower. "
+ + "Current channel leader is {}",
+ lookupServiceAddress,
+ leaderElectionService.getCurrentLeader());
+ }
+ }
+ });
+ leaderElectionService.start();
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(SCHEMA)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(SCHEMA)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ // TODO: schedule cleanupTasks by brokerRegistry
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+
+ public void close() throws PulsarServerException {
+ try {
+ if (leaderElectionService != null) {
+ leaderElectionService.close();
+ leaderElectionService = null;
+ log.debug("Successfully closed the channel leader election service.");
+ }
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public void scheduleCompaction() throws PulsarServerException {
+ try {
+ Long threshold = pulsar.getAdminClient().topicPolicies()
+ .getCompactionThreshold(TOPIC);
+ if (threshold == null || threshold == 0) {
+ pulsar.getAdminClient().topicPolicies()
+ .setCompactionThreshold(TOPIC, COMPACTION_THRESHOLD);
+ log.info("Scheduled compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ } else {
+ log.info("Already set compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ }
+ } catch (PulsarAdminException e) {
+ throw new PulsarServerException("Failed to schedule compaction.", e);
+ }
+ }
+
+ public String getChannelOwner() {
+ CompletableFuture<Optional<LeaderBroker>> future = leaderElectionService.readCurrentLeader();
+ if (!future.isDone()) {
+ return null;
+ }
+ Optional<LeaderBroker> leader = future.join();
+ if (leader.isEmpty()) {
+ return null;
+ }
+ //expecting http://broker-xyz:port
+ String broker = leader.get().getServiceUrl();
+ broker = broker.substring(broker.lastIndexOf('/') + 1);
+ return broker;
+ }
+
+ public boolean isChannelOwner() {
+ return isTargetBroker(getChannelOwner());
+ }
+
+ public CompletableFuture<String> getOwnerAsync(String serviceUnit) {
+
+ ServiceUnitStateData data = tableview.get(serviceUnit);
+ if (data == null) {
+ return null;
+ }
+ switch (data.state()) {
+ case Owned, Splitting -> {
+ return CompletableFuture.completedFuture(data.broker());
+ }
+ case Assigned, Released -> {
+ return deferGetOwnerRequest(serviceUnit);
+ }
+ default -> {
+ return null;
+ }
+ }
+ }
+
+ public CompletableFuture<String> publishAssignEventAsync(String serviceUnit, String broker) {
+ CompletableFuture<String> getOwnerRequest = deferGetOwnerRequest(serviceUnit);
+ pubAsync(serviceUnit, new ServiceUnitStateData(Assigned, broker))
+ .whenComplete((__, ex) -> {
+ if (ex != null) {
+ getOwnerRequests.remove(serviceUnit);
+ if (!getOwnerRequest.isCompletedExceptionally()) {
+ getOwnerRequest.completeExceptionally(ex);
+ }
+ }
+ });
+
+ return getOwnerRequest;
+ }
+
+ public CompletableFuture<Void> publishUnloadEventAsync(Unload unload) {
+ String serviceUnit = unload.serviceUnit();
+ if (isTransferCommand(unload)) {
+ ServiceUnitStateData next = new ServiceUnitStateData(Assigned,
+ unload.destBroker().get(), unload.sourceBroker());
+ return pubAsync(serviceUnit, next).thenAccept(__ -> {
+ });
+ }
+ return tombstoneAsync(serviceUnit).thenAccept(__ -> {
+ });
+ }
+
+ public CompletableFuture<Void> publishSplitEventAsync(Split split) {
+ String serviceUnit = split.serviceUnit();
+ ServiceUnitStateData data = tableview.get(serviceUnit);
+ ServiceUnitStateData next = new ServiceUnitStateData(Splitting, data.broker());
+ return pubAsync(serviceUnit, next).thenAccept(__ -> {
+ });
+ }
+
+ private void handle(String serviceUnit, ServiceUnitStateData data) {
+ if (log.isDebugEnabled()) {
+ log.info("{} received a handle request for serviceUnit:{}, data:{}",
+ lookupServiceAddress, serviceUnit, data);
+ }
+
+ if (data == null) {
+ handleTombstoneEvent(serviceUnit);
+ return;
+ }
+
+ // TODO : Add state validation
+ switch (data.state()) {
+ case Owned -> handleOwnEvent(serviceUnit, data);
+ case Assigned -> handleAssignEvent(serviceUnit, data);
+ case Released -> handleReleaseEvent(serviceUnit, data);
+ case Splitting -> handleSplitEvent(serviceUnit, data);
+ default -> throw new IllegalStateException("Failed to handle channel data:" + data);
+ }
+ }
+
+ private static boolean isTransferCommand(ServiceUnitStateData data) {
+ if (data == null) {
+ return false;
+ }
+ return StringUtils.isNotEmpty(data.sourceBroker());
+ }
+
+ private static boolean isTransferCommand(Unload data) {
+ return data.destBroker().isPresent();
+ }
+
+ private static String getLogEventTag(ServiceUnitStateData data) {
+ return data == null ? "Free" :
+ isTransferCommand(data) ? "Transfer:" + data.state() : data.state().toString();
+ }
+
+ private void log(Throwable e, String serviceUnit, ServiceUnitStateData data, ServiceUnitStateData next) {
+ if (e == null) {
+ if (log.isDebugEnabled() || isTransferCommand(data)) {
+ log.info("{} handled {} event for serviceUnit:{}, cur:{}, next:{}",
+ lookupServiceAddress, getLogEventTag(data), serviceUnit,
+ data == null ? "" : data,
+ next == null ? "" : next);
+ }
+ } else {
+ log.error("{} failed to handle {} event for serviceUnit:{}, cur:{}, next:{}",
+ lookupServiceAddress, getLogEventTag(data), serviceUnit,
+ data == null ? "" : data,
+ next == null ? "" : next,
+ e);
+ }
+ }
+
+ private void handleOwnEvent(String serviceUnit, ServiceUnitStateData data) {
+ var getOwnerRequest = getOwnerRequests.remove(serviceUnit);
+ if (getOwnerRequest != null) {
+ getOwnerRequest.complete(data.broker());
+ }
+ if (isTargetBroker(data.broker())) {
+ log(null, serviceUnit, data, null);
+ }
+
+ }
+
+ private void handleAssignEvent(String serviceUnit, ServiceUnitStateData data) {
+ deferGetOwnerRequest(serviceUnit);
+ if (isTargetBroker(data.broker())) {
+ ServiceUnitStateData next = new ServiceUnitStateData(
+ isTransferCommand(data) ? Released : Owned, data.broker(), data.sourceBroker());
+ pubAsync(serviceUnit, next)
+ .whenComplete((__, e) -> log(e, serviceUnit, data, next));
+ }
+ }
+
+ private void handleReleaseEvent(String serviceUnit, ServiceUnitStateData data) {
+
+ if (isTargetBroker(data.sourceBroker())) {
+ ServiceUnitStateData next = new ServiceUnitStateData(Owned, data.broker(), data.sourceBroker());
+ // TODO: when close, pass message to clients to connect to the new broker
+ closeServiceUnit(serviceUnit)
+ .thenCompose(__ -> pubAsync(serviceUnit, next))
+ .whenComplete((__, e) -> log(e, serviceUnit, data, next));
+ }
+ }
+
+ private void handleSplitEvent(String serviceUnit, ServiceUnitStateData data) {
+ if (isTargetBroker(data.broker())) {
+ splitServiceUnit(serviceUnit)
+ .thenCompose(__ -> tombstoneAsync(serviceUnit))
+ .whenComplete((__, e) -> log(e, serviceUnit, data, null));
+ }
+ }
+
+ private void handleTombstoneEvent(String serviceUnit) {
+ closeServiceUnit(serviceUnit)
+ .thenAccept(__ -> {
+ var request = getOwnerRequests.remove(serviceUnit);
+ if (request != null) {
+ request.completeExceptionally(new IllegalStateException("The ownership has been unloaded. "
+ + "No owner is found for serviceUnit: " + serviceUnit));
+ }
+ })
+ .whenComplete((__, e) -> log(e, serviceUnit, null, null));
+ }
+
+ private CompletableFuture<MessageId> pubAsync(String serviceUnit, ServiceUnitStateData data) {
+ CompletableFuture<MessageId> future = new CompletableFuture<>();
+ try {
+ outstandingPubMessages.acquire();
+ } catch (InterruptedException e) {
+ log.warn("Interrupted while acquiring semaphore to publish a message for serviceUnit:{}, data:{}",
+ serviceUnit, data);
+ future.completeExceptionally(e);
+ return future;
+ }
+ producer.newMessage()
+ .key(serviceUnit)
+ .value(data)
+ .sendAsync()
+ .whenComplete((messageId, e) -> {
+ outstandingPubMessages.release();
+ if (e != null) {
+ future.completeExceptionally(e);
+ } else {
+ future.complete(messageId);
+ }
+ });
+ return future;
+ }
+
+ private CompletableFuture<MessageId> tombstoneAsync(String serviceUnit) {
+ return pubAsync(serviceUnit, null);
+ }
+
+ private boolean isTargetBroker(String broker) {
+ if (broker == null) {
+ return false;
+ }
+ // TODO: remove broker port from the input broker
Review Comment:
Removed the comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] heesung-sn commented on a diff in pull request #18489: [improve][broker] PIP-192 Added ServiceUnitStateChannelImpl
Posted by GitBox <gi...@apache.org>.
heesung-sn commented on code in PR #18489:
URL: https://github.com/apache/pulsar/pull/18489#discussion_r1034421259
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,778 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Free;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Closed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Constructed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.LeaderElectionServiceStarted;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Started;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private final Schema<ServiceUnitStateData> schema;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private final String lookupServiceAddress;
+ // TODO: define BrokerRegistry
+ private final Semaphore outstandingCleanupTombstoneMessages;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+ private final LeaderElectionService leaderElectionService;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private ScheduledFuture<?> cleanupTasks;
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+ private volatile ChannelState channelState;
+
+ enum ChannelState {
+ Closed(0),
+ Constructed(1),
+ LeaderElectionServiceStarted(2),
+ Started(3);
+
+ ChannelState(int id) {
+ this.id = id;
+ }
+ int id;
+ }
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ this.lookupServiceAddress = pulsar.getLookupServiceAddress();
+ this.schema = Schema.JSON(ServiceUnitStateData.class);
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder().build();
+ this.cleanupJobs = ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder().build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} is the leader now.", lookupServiceAddress);
+ // TODO: schedule monitorOwnerships by brokerRegistry
+ } else {
+ log.debug("This broker:{} is a follower now.", lookupServiceAddress);
+ // TODO: cancel scheduled monitorOwnerships if any
+ }
+ });
+ this.channelState = Constructed;
+ }
+
+ public synchronized void start() throws PulsarServerException {
+ validateChannelState(LeaderElectionServiceStarted, false);
+ try {
+ leaderElectionService.start();
+ this.channelState = LeaderElectionServiceStarted;
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(schema)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(schema)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ channelState = Started;
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public synchronized void close() throws PulsarServerException {
+ channelState = Closed;
+ try {
+ leaderElectionService.close();
+ log.debug("Successfully closed the channel leader election service.");
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ private void validateChannelState(ChannelState targetState, boolean checkLowerIds) {
+ int order = checkLowerIds ? -1 : 1;
+ if (Integer.compare(channelState.id, targetState.id) * order > 0) {
+ throw new IllegalStateException("Invalid channel state:" + channelState.name());
+ }
+ }
+
+ public synchronized void scheduleCompaction() throws PulsarServerException {
+ try {
+ Long threshold = pulsar.getAdminClient().topicPolicies()
+ .getCompactionThreshold(TOPIC);
+ if (threshold == null || threshold == 0) {
+ pulsar.getAdminClient().topicPolicies()
+ .setCompactionThreshold(TOPIC, COMPACTION_THRESHOLD);
+ log.info("Scheduled compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ } else {
+ log.info("Already set compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
Review Comment:
removed this compaction schedule api as not required.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] heesung-sn commented on a diff in pull request #18489: [improve][broker] PIP-192 Added ServiceUnitStateChannelImpl
Posted by GitBox <gi...@apache.org>.
heesung-sn commented on code in PR #18489:
URL: https://github.com/apache/pulsar/pull/18489#discussion_r1034310124
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,778 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Free;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Closed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Constructed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.LeaderElectionServiceStarted;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Started;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private final Schema<ServiceUnitStateData> schema;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private final String lookupServiceAddress;
+ // TODO: define BrokerRegistry
+ private final Semaphore outstandingCleanupTombstoneMessages;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+ private final LeaderElectionService leaderElectionService;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private ScheduledFuture<?> cleanupTasks;
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+ private volatile ChannelState channelState;
+
+ enum ChannelState {
+ Closed(0),
+ Constructed(1),
+ LeaderElectionServiceStarted(2),
+ Started(3);
+
+ ChannelState(int id) {
+ this.id = id;
+ }
+ int id;
+ }
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ this.lookupServiceAddress = pulsar.getLookupServiceAddress();
+ this.schema = Schema.JSON(ServiceUnitStateData.class);
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder().build();
+ this.cleanupJobs = ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder().build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} is the leader now.", lookupServiceAddress);
+ // TODO: schedule monitorOwnerships by brokerRegistry
+ } else {
+ log.debug("This broker:{} is a follower now.", lookupServiceAddress);
+ // TODO: cancel scheduled monitorOwnerships if any
+ }
+ });
+ this.channelState = Constructed;
+ }
+
+ public synchronized void start() throws PulsarServerException {
Review Comment:
Yes, but the channelOwner will own the system topic.(The leaderElectionServiceStarted channel state can return getChannelOwner api.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] heesung-sn commented on a diff in pull request #18489: [improve][broker] PIP-192 Added ServiceUnitStateChannelImpl
Posted by GitBox <gi...@apache.org>.
heesung-sn commented on code in PR #18489:
URL: https://github.com/apache/pulsar/pull/18489#discussion_r1034422109
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,778 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Free;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Closed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Constructed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.LeaderElectionServiceStarted;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Started;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private final Schema<ServiceUnitStateData> schema;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private final String lookupServiceAddress;
+ // TODO: define BrokerRegistry
+ private final Semaphore outstandingCleanupTombstoneMessages;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+ private final LeaderElectionService leaderElectionService;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private ScheduledFuture<?> cleanupTasks;
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+ private volatile ChannelState channelState;
+
+ enum ChannelState {
+ Closed(0),
+ Constructed(1),
+ LeaderElectionServiceStarted(2),
+ Started(3);
+
+ ChannelState(int id) {
+ this.id = id;
+ }
+ int id;
+ }
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ this.lookupServiceAddress = pulsar.getLookupServiceAddress();
+ this.schema = Schema.JSON(ServiceUnitStateData.class);
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder().build();
+ this.cleanupJobs = ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder().build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} is the leader now.", lookupServiceAddress);
+ // TODO: schedule monitorOwnerships by brokerRegistry
+ } else {
+ log.debug("This broker:{} is a follower now.", lookupServiceAddress);
+ // TODO: cancel scheduled monitorOwnerships if any
+ }
+ });
+ this.channelState = Constructed;
+ }
+
+ public synchronized void start() throws PulsarServerException {
+ validateChannelState(LeaderElectionServiceStarted, false);
+ try {
+ leaderElectionService.start();
+ this.channelState = LeaderElectionServiceStarted;
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(schema)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(schema)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ channelState = Started;
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public synchronized void close() throws PulsarServerException {
+ channelState = Closed;
+ try {
+ leaderElectionService.close();
+ log.debug("Successfully closed the channel leader election service.");
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ private void validateChannelState(ChannelState targetState, boolean checkLowerIds) {
+ int order = checkLowerIds ? -1 : 1;
+ if (Integer.compare(channelState.id, targetState.id) * order > 0) {
+ throw new IllegalStateException("Invalid channel state:" + channelState.name());
+ }
+ }
+
+ public synchronized void scheduleCompaction() throws PulsarServerException {
+ try {
+ Long threshold = pulsar.getAdminClient().topicPolicies()
+ .getCompactionThreshold(TOPIC);
+ if (threshold == null || threshold == 0) {
+ pulsar.getAdminClient().topicPolicies()
+ .setCompactionThreshold(TOPIC, COMPACTION_THRESHOLD);
+ log.info("Scheduled compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ } else {
+ log.info("Already set compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ }
+ } catch (PulsarAdminException e) {
+ throw new PulsarServerException("Failed to schedule compaction.", e);
+ }
+ }
+
+ public String getChannelOwner() {
+ validateChannelState(LeaderElectionServiceStarted, true);
+ CompletableFuture<Optional<LeaderBroker>> future = leaderElectionService.readCurrentLeader();
+ if (!future.isDone()) {
+ return null;
+ }
+ Optional<LeaderBroker> leader = future.join();
+ if (leader.isEmpty()) {
+ return null;
+ }
+ //expecting http://broker-xyz:port
+ // TODO: discard this protocol prefix removal by a util func that returns lookupServiceAddress(serviceUrl)
+ String broker = leader.get().getServiceUrl();
+ broker = broker.substring(broker.lastIndexOf('/') + 1);
+ return broker;
+ }
+
+ public boolean isChannelOwner() {
+ return isTargetBroker(getChannelOwner());
+ }
+
+ public CompletableFuture<String> getOwnerAsync(String serviceUnit) {
+ validateChannelState(Started, true);
+ ServiceUnitStateData data = tableview.get(serviceUnit);
+ if (data == null) {
+ return null;
Review Comment:
updated.
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,778 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Free;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Closed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Constructed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.LeaderElectionServiceStarted;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Started;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private final Schema<ServiceUnitStateData> schema;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private final String lookupServiceAddress;
+ // TODO: define BrokerRegistry
+ private final Semaphore outstandingCleanupTombstoneMessages;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+ private final LeaderElectionService leaderElectionService;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private ScheduledFuture<?> cleanupTasks;
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+ private volatile ChannelState channelState;
+
+ enum ChannelState {
+ Closed(0),
+ Constructed(1),
+ LeaderElectionServiceStarted(2),
+ Started(3);
+
+ ChannelState(int id) {
+ this.id = id;
+ }
+ int id;
+ }
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ this.lookupServiceAddress = pulsar.getLookupServiceAddress();
+ this.schema = Schema.JSON(ServiceUnitStateData.class);
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder().build();
+ this.cleanupJobs = ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder().build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} is the leader now.", lookupServiceAddress);
+ // TODO: schedule monitorOwnerships by brokerRegistry
+ } else {
+ log.debug("This broker:{} is a follower now.", lookupServiceAddress);
+ // TODO: cancel scheduled monitorOwnerships if any
+ }
+ });
+ this.channelState = Constructed;
+ }
+
+ public synchronized void start() throws PulsarServerException {
+ validateChannelState(LeaderElectionServiceStarted, false);
+ try {
+ leaderElectionService.start();
+ this.channelState = LeaderElectionServiceStarted;
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(schema)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(schema)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ channelState = Started;
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public synchronized void close() throws PulsarServerException {
+ channelState = Closed;
+ try {
+ leaderElectionService.close();
+ log.debug("Successfully closed the channel leader election service.");
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ private void validateChannelState(ChannelState targetState, boolean checkLowerIds) {
+ int order = checkLowerIds ? -1 : 1;
+ if (Integer.compare(channelState.id, targetState.id) * order > 0) {
+ throw new IllegalStateException("Invalid channel state:" + channelState.name());
+ }
+ }
+
+ public synchronized void scheduleCompaction() throws PulsarServerException {
+ try {
+ Long threshold = pulsar.getAdminClient().topicPolicies()
+ .getCompactionThreshold(TOPIC);
+ if (threshold == null || threshold == 0) {
+ pulsar.getAdminClient().topicPolicies()
+ .setCompactionThreshold(TOPIC, COMPACTION_THRESHOLD);
+ log.info("Scheduled compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ } else {
+ log.info("Already set compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ }
+ } catch (PulsarAdminException e) {
+ throw new PulsarServerException("Failed to schedule compaction.", e);
+ }
+ }
+
+ public String getChannelOwner() {
Review Comment:
Make sense. updated.
##########
pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelTest.java:
##########
@@ -0,0 +1,796 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.ConnectionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.Reconnected;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.greaterThanOrEqualTo;
+import static org.hamcrest.Matchers.lessThanOrEqualTo;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.assertNull;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.eq;
+import static org.mockito.Mockito.doReturn;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.spy;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.testng.Assert.assertFalse;
+import static org.testng.Assert.assertTrue;
+import static org.testng.AssertJUnit.assertNotNull;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import org.apache.commons.lang.StringUtils;
+import org.apache.commons.lang3.reflect.FieldUtils;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.auth.MockedPulsarServiceBaseTest;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.TypedMessageBuilder;
+import org.apache.pulsar.client.impl.TableViewImpl;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+import org.awaitility.Awaitility;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.Test;
+
+@Test(groups = "broker")
+public class ServiceUnitStateChannelTest extends MockedPulsarServiceBaseTest {
+
+ private PulsarService pulsar1;
+ private PulsarService pulsar2;
+ private ServiceUnitStateChannel channel1;
+ private ServiceUnitStateChannel channel2;
+ private String lookupServiceAddress1;
+ private String lookupServiceAddress2;
+ private String bundle;
+
+ private String bundle1;
+ private String bundle2;
+
+ @BeforeClass
+ @Override
+ protected void setup() throws Exception {
+ conf.setAllowAutoTopicCreation(true);
+ super.internalSetup(conf);
+
+ admin.tenants().createTenant("pulsar", createDefaultTenantInfo());
+ admin.namespaces().createNamespace("pulsar/system");
+ admin.tenants().createTenant("public", createDefaultTenantInfo());
+ admin.namespaces().createNamespace("public/default");
+
+ pulsar1 = pulsar;
+ pulsar2 = startBrokerWithoutAuthorization(getDefaultConf());
+ channel1 = new ServiceUnitStateChannelImpl(pulsar1);
+ channel1.start();
+ channel2 = new ServiceUnitStateChannelImpl(pulsar2);
+ channel2.start();
+ lookupServiceAddress1 = (String)
+ FieldUtils.readDeclaredField(channel1, "lookupServiceAddress", true);
+ lookupServiceAddress2 = (String)
+ FieldUtils.readDeclaredField(channel2, "lookupServiceAddress", true);
+
+ bundle = String.format("%s/%s", "public/default", "0x00000000_0xffffffff");
+ bundle1 = String.format("%s/%s", "public/default", "0x00000000_0xfffffff0");
+ bundle2 = String.format("%s/%s", "public/default", "0xfffffff0_0xffffffff");
+ }
+
+ @BeforeMethod
+ protected void initTableViews() throws Exception {
+ cleanTableView(channel1, bundle);
+ cleanTableView(channel2, bundle);
+ }
+
+
+ @AfterClass
+ @Override
+ protected void cleanup() throws Exception {
+ channel1.close();
+ channel2.close();
+ pulsar1 = null;
+ pulsar2.close();
+ super.internalCleanup();
+ }
+
+ @Test(priority = 0)
+ public void channelOwnerTest() throws Exception {
+ var channelOwner1 = channel1.getChannelOwner();
+ var channelOwner2 = channel2.getChannelOwner();
+ assertEquals(channelOwner1, channelOwner2);
+ LeaderElectionService leaderElectionService1 = (LeaderElectionService) FieldUtils.readDeclaredField(
+ channel1, "leaderElectionService", true);
+ leaderElectionService1.close();
+ waitUntilNewChannelOwner(channel2, channelOwner1);
+ leaderElectionService1.start();
+ waitUntilNewChannelOwner(channel1, channelOwner1);
+
+ var newChannelOwner1 = channel1.getChannelOwner();
+ var newChannelOwner2 = channel2.getChannelOwner();
+
+ assertEquals(newChannelOwner1, newChannelOwner2);
+ assertNotEquals(channelOwner1, newChannelOwner1);
+
+ if (newChannelOwner1.equals(lookupServiceAddress1)) {
+ assertTrue(channel1.isChannelOwner());
+ assertFalse(channel2.isChannelOwner());
+ } else {
+ assertFalse(channel1.isChannelOwner());
+ assertTrue(channel2.isChannelOwner());
+ }
+ }
+
+ @Test(priority = 0)
+ public void channelValidationTest()
+ throws ExecutionException, InterruptedException, IllegalAccessException, PulsarServerException {
+ var channel = new ServiceUnitStateChannelImpl(pulsar);
+ int errorCnt = validateChannelStart(channel);
+ assertEquals(6, errorCnt);
+ ExecutorService executor = Executors.newSingleThreadExecutor();
+ Future startFuture = executor.submit(() -> {
+ try {
+ channel.start();
+ } catch (PulsarServerException e) {
+ throw new RuntimeException(e);
+ }
+ });
+ errorCnt = validateChannelStart(channel);
+ startFuture.get();
+ assertTrue(errorCnt > 0);
+
+ FieldUtils.writeDeclaredField(channel, "channelState",
+ ServiceUnitStateChannelImpl.ChannelState.LeaderElectionServiceStarted, true);
+ assertNotNull(channel.getChannelOwner());
+
+ Future closeFuture = executor.submit(()->{
+ try {
+ channel.close();
+ } catch (PulsarServerException e) {
+ throw new RuntimeException(e);
+ }
+ });
+ errorCnt = validateChannelStart(channel);
+ closeFuture.get();
+ assertTrue(errorCnt > 0);
+ errorCnt = validateChannelStart(channel);
+ assertEquals(6, errorCnt);
+
+ // check if we can close() again
+ channel.close();
+ errorCnt = validateChannelStart(channel);
+ assertEquals(6, errorCnt);
+
+ // close() -> start() test.
+ channel.start();
+ assertNotNull(channel.getChannelOwner());
+
+ // start() -> start() test.
+ IllegalStateException ex = null;
Review Comment:
updated.
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,778 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Free;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Closed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Constructed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.LeaderElectionServiceStarted;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Started;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private final Schema<ServiceUnitStateData> schema;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private final String lookupServiceAddress;
+ // TODO: define BrokerRegistry
+ private final Semaphore outstandingCleanupTombstoneMessages;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+ private final LeaderElectionService leaderElectionService;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private ScheduledFuture<?> cleanupTasks;
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+ private volatile ChannelState channelState;
+
+ enum ChannelState {
+ Closed(0),
+ Constructed(1),
+ LeaderElectionServiceStarted(2),
+ Started(3);
+
+ ChannelState(int id) {
+ this.id = id;
+ }
+ int id;
+ }
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ this.lookupServiceAddress = pulsar.getLookupServiceAddress();
+ this.schema = Schema.JSON(ServiceUnitStateData.class);
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder().build();
+ this.cleanupJobs = ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder().build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} is the leader now.", lookupServiceAddress);
+ // TODO: schedule monitorOwnerships by brokerRegistry
+ } else {
+ log.debug("This broker:{} is a follower now.", lookupServiceAddress);
+ // TODO: cancel scheduled monitorOwnerships if any
+ }
+ });
+ this.channelState = Constructed;
+ }
+
+ public synchronized void start() throws PulsarServerException {
+ validateChannelState(LeaderElectionServiceStarted, false);
+ try {
+ leaderElectionService.start();
+ this.channelState = LeaderElectionServiceStarted;
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(schema)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(schema)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ channelState = Started;
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public synchronized void close() throws PulsarServerException {
+ channelState = Closed;
+ try {
+ leaderElectionService.close();
+ log.debug("Successfully closed the channel leader election service.");
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ private void validateChannelState(ChannelState targetState, boolean checkLowerIds) {
+ int order = checkLowerIds ? -1 : 1;
+ if (Integer.compare(channelState.id, targetState.id) * order > 0) {
+ throw new IllegalStateException("Invalid channel state:" + channelState.name());
+ }
+ }
+
+ public synchronized void scheduleCompaction() throws PulsarServerException {
+ try {
+ Long threshold = pulsar.getAdminClient().topicPolicies()
+ .getCompactionThreshold(TOPIC);
+ if (threshold == null || threshold == 0) {
+ pulsar.getAdminClient().topicPolicies()
+ .setCompactionThreshold(TOPIC, COMPACTION_THRESHOLD);
+ log.info("Scheduled compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ } else {
+ log.info("Already set compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ }
+ } catch (PulsarAdminException e) {
+ throw new PulsarServerException("Failed to schedule compaction.", e);
+ }
+ }
+
+ public String getChannelOwner() {
+ validateChannelState(LeaderElectionServiceStarted, true);
+ CompletableFuture<Optional<LeaderBroker>> future = leaderElectionService.readCurrentLeader();
+ if (!future.isDone()) {
+ return null;
+ }
+ Optional<LeaderBroker> leader = future.join();
+ if (leader.isEmpty()) {
+ return null;
+ }
+ //expecting http://broker-xyz:port
+ // TODO: discard this protocol prefix removal by a util func that returns lookupServiceAddress(serviceUrl)
+ String broker = leader.get().getServiceUrl();
+ broker = broker.substring(broker.lastIndexOf('/') + 1);
+ return broker;
+ }
+
+ public boolean isChannelOwner() {
+ return isTargetBroker(getChannelOwner());
+ }
+
+ public CompletableFuture<String> getOwnerAsync(String serviceUnit) {
+ validateChannelState(Started, true);
+ ServiceUnitStateData data = tableview.get(serviceUnit);
+ if (data == null) {
+ return null;
+ }
+ switch (data.state()) {
+ case Owned, Splitting -> {
+ return CompletableFuture.completedFuture(data.broker());
+ }
+ case Assigned, Released -> {
+ return deferGetOwnerRequest(serviceUnit);
+ }
+ default -> {
+ return null;
+ }
+ }
+ }
+
+ public CompletableFuture<String> publishAssignEventAsync(String serviceUnit, String broker) {
+ CompletableFuture<String> getOwnerRequest = deferGetOwnerRequest(serviceUnit);
+ pubAsync(serviceUnit, new ServiceUnitStateData(Assigned, broker))
+ .whenComplete((__, ex) -> {
+ if (ex != null) {
+ getOwnerRequests.remove(serviceUnit);
+ if (!getOwnerRequest.isCompletedExceptionally()) {
+ getOwnerRequest.completeExceptionally(ex);
+ }
+ }
+ });
+
+ return getOwnerRequest;
+ }
+
+ public CompletableFuture<Void> publishUnloadEventAsync(Unload unload) {
+ String serviceUnit = unload.serviceUnit();
+ if (isTransferCommand(unload)) {
+ ServiceUnitStateData next = new ServiceUnitStateData(Assigned,
+ unload.destBroker().get(), unload.sourceBroker());
+ return pubAsync(serviceUnit, next).thenAccept(__ -> {
+ });
+ }
+ return tombstoneAsync(serviceUnit).thenAccept(__ -> {
+ });
+ }
+
+ public CompletableFuture<Void> publishSplitEventAsync(Split split) {
+ String serviceUnit = split.serviceUnit();
+ ServiceUnitStateData next = new ServiceUnitStateData(Splitting, split.sourceBroker());
+ return pubAsync(serviceUnit, next).thenAccept(__ -> {
+ });
+ }
+
+ private void handle(String serviceUnit, ServiceUnitStateData data) {
+ if (log.isDebugEnabled()) {
+ log.info("{} received a handle request for serviceUnit:{}, data:{}",
+ lookupServiceAddress, serviceUnit, data);
+ }
+
+ ServiceUnitState state = data == null ? Free : data.state();
+
+ // TODO : Add state validation in tableview by the compaction strategy
+ switch (state) {
+ case Owned -> handleOwnEvent(serviceUnit, data);
+ case Assigned -> handleAssignEvent(serviceUnit, data);
+ case Released -> handleReleaseEvent(serviceUnit, data);
+ case Splitting -> handleSplitEvent(serviceUnit, data);
+ case Free -> handleFreeEvent(serviceUnit);
+ default -> throw new IllegalStateException("Failed to handle channel data:" + data);
+ }
+ }
+
+ private static boolean isTransferCommand(ServiceUnitStateData data) {
+ if (data == null) {
+ return false;
+ }
+ return StringUtils.isNotEmpty(data.sourceBroker());
+ }
+
+ private static boolean isTransferCommand(Unload data) {
+ return data.destBroker().isPresent();
+ }
+
+ private static String getLogEventTag(ServiceUnitStateData data) {
+ return data == null ? "Free" :
+ isTransferCommand(data) ? "Transfer:" + data.state() : data.state().toString();
+ }
+
+ private void log(Throwable e, String serviceUnit, ServiceUnitStateData data, ServiceUnitStateData next) {
+ if (e == null) {
+ if (log.isDebugEnabled() || isTransferCommand(data)) {
+ log.info("{} handled {} event for serviceUnit:{}, cur:{}, next:{}",
+ lookupServiceAddress, getLogEventTag(data), serviceUnit,
+ data == null ? "" : data,
+ next == null ? "" : next);
+ }
+ } else {
+ log.error("{} failed to handle {} event for serviceUnit:{}, cur:{}, next:{}",
+ lookupServiceAddress, getLogEventTag(data), serviceUnit,
+ data == null ? "" : data,
+ next == null ? "" : next,
+ e);
+ }
+ }
+
+ private void handleOwnEvent(String serviceUnit, ServiceUnitStateData data) {
+ var getOwnerRequest = getOwnerRequests.remove(serviceUnit);
+ if (getOwnerRequest != null) {
+ getOwnerRequest.complete(data.broker());
+ }
+ if (isTargetBroker(data.broker())) {
+ log(null, serviceUnit, data, null);
+ }
+
+ }
+
+ private void handleAssignEvent(String serviceUnit, ServiceUnitStateData data) {
+ deferGetOwnerRequest(serviceUnit);
+ if (isTargetBroker(data.broker())) {
+ ServiceUnitStateData next = new ServiceUnitStateData(
+ isTransferCommand(data) ? Released : Owned, data.broker(), data.sourceBroker());
+ pubAsync(serviceUnit, next)
+ .whenComplete((__, e) -> log(e, serviceUnit, data, next));
+ }
+ }
+
+ private void handleReleaseEvent(String serviceUnit, ServiceUnitStateData data) {
+
+ if (isTargetBroker(data.sourceBroker())) {
+ ServiceUnitStateData next = new ServiceUnitStateData(Owned, data.broker(), data.sourceBroker());
+ // TODO: when close, pass message to clients to connect to the new broker
+ closeServiceUnit(serviceUnit)
+ .thenCompose(__ -> pubAsync(serviceUnit, next))
+ .whenComplete((__, e) -> log(e, serviceUnit, data, next));
+ }
+ }
+
+ private void handleSplitEvent(String serviceUnit, ServiceUnitStateData data) {
+ if (isTargetBroker(data.broker())) {
+ splitServiceUnit(serviceUnit)
+ .thenCompose(__ -> tombstoneAsync(serviceUnit))
+ .whenComplete((__, e) -> log(e, serviceUnit, data, null));
+ }
+ }
+
+ private void handleFreeEvent(String serviceUnit) {
+ closeServiceUnit(serviceUnit)
+ .thenAccept(__ -> {
+ var request = getOwnerRequests.remove(serviceUnit);
+ if (request != null) {
+ request.completeExceptionally(new IllegalStateException("The ownership has been unloaded. "
+ + "No owner is found for serviceUnit: " + serviceUnit));
+ }
+ })
+ .whenComplete((__, e) -> log(e, serviceUnit, null, null));
+ }
+
+ private CompletableFuture<MessageId> pubAsync(String serviceUnit, ServiceUnitStateData data) {
+ validateChannelState(Started, true);
+ CompletableFuture<MessageId> future = new CompletableFuture<>();
+ producer.newMessage()
+ .key(serviceUnit)
+ .value(data)
+ .sendAsync()
+ .whenComplete((messageId, e) -> {
+ if (e != null) {
+ log.error("Failed to publish the message: serviceUnit:{}, data:{}",
+ serviceUnit, data, e);
+ future.completeExceptionally(e);
+ } else {
+ future.complete(messageId);
+ }
+ });
+ return future;
+ }
+
+ private CompletableFuture<MessageId> tombstoneAsync(String serviceUnit) {
+ return pubAsync(serviceUnit, null);
+ }
+
+ private boolean isTargetBroker(String broker) {
+ if (broker == null) {
+ return false;
+ }
+ return broker.equals(lookupServiceAddress);
+ }
+
+ private NamespaceBundle getNamespaceBundle(String bundle) {
+ final String namespaceName = LoadManagerShared.getNamespaceNameFromBundleName(bundle);
+ final String bundleRange = LoadManagerShared.getBundleRangeFromBundleName(bundle);
+ return pulsar.getNamespaceService().getNamespaceBundleFactory().getBundle(namespaceName, bundleRange);
+ }
+
+ private CompletableFuture<String> deferGetOwnerRequest(String serviceUnit) {
+ return getOwnerRequests
+ .computeIfAbsent(serviceUnit, k -> {
+ CompletableFuture future = new CompletableFuture<>();
Review Comment:
updated.
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,778 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Free;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Closed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Constructed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.LeaderElectionServiceStarted;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Started;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private final Schema<ServiceUnitStateData> schema;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private final String lookupServiceAddress;
+ // TODO: define BrokerRegistry
+ private final Semaphore outstandingCleanupTombstoneMessages;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+ private final LeaderElectionService leaderElectionService;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private ScheduledFuture<?> cleanupTasks;
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+ private volatile ChannelState channelState;
+
+ enum ChannelState {
+ Closed(0),
+ Constructed(1),
+ LeaderElectionServiceStarted(2),
+ Started(3);
+
+ ChannelState(int id) {
+ this.id = id;
+ }
+ int id;
+ }
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ this.lookupServiceAddress = pulsar.getLookupServiceAddress();
+ this.schema = Schema.JSON(ServiceUnitStateData.class);
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder().build();
+ this.cleanupJobs = ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder().build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} is the leader now.", lookupServiceAddress);
+ // TODO: schedule monitorOwnerships by brokerRegistry
+ } else {
+ log.debug("This broker:{} is a follower now.", lookupServiceAddress);
+ // TODO: cancel scheduled monitorOwnerships if any
+ }
+ });
+ this.channelState = Constructed;
+ }
+
+ public synchronized void start() throws PulsarServerException {
+ validateChannelState(LeaderElectionServiceStarted, false);
+ try {
+ leaderElectionService.start();
+ this.channelState = LeaderElectionServiceStarted;
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(schema)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(schema)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ channelState = Started;
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public synchronized void close() throws PulsarServerException {
+ channelState = Closed;
+ try {
+ leaderElectionService.close();
+ log.debug("Successfully closed the channel leader election service.");
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ private void validateChannelState(ChannelState targetState, boolean checkLowerIds) {
+ int order = checkLowerIds ? -1 : 1;
+ if (Integer.compare(channelState.id, targetState.id) * order > 0) {
+ throw new IllegalStateException("Invalid channel state:" + channelState.name());
+ }
+ }
+
+ public synchronized void scheduleCompaction() throws PulsarServerException {
+ try {
+ Long threshold = pulsar.getAdminClient().topicPolicies()
+ .getCompactionThreshold(TOPIC);
+ if (threshold == null || threshold == 0) {
+ pulsar.getAdminClient().topicPolicies()
+ .setCompactionThreshold(TOPIC, COMPACTION_THRESHOLD);
+ log.info("Scheduled compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ } else {
+ log.info("Already set compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ }
+ } catch (PulsarAdminException e) {
+ throw new PulsarServerException("Failed to schedule compaction.", e);
+ }
+ }
+
+ public String getChannelOwner() {
+ validateChannelState(LeaderElectionServiceStarted, true);
+ CompletableFuture<Optional<LeaderBroker>> future = leaderElectionService.readCurrentLeader();
+ if (!future.isDone()) {
+ return null;
+ }
+ Optional<LeaderBroker> leader = future.join();
+ if (leader.isEmpty()) {
+ return null;
+ }
+ //expecting http://broker-xyz:port
+ // TODO: discard this protocol prefix removal by a util func that returns lookupServiceAddress(serviceUrl)
+ String broker = leader.get().getServiceUrl();
+ broker = broker.substring(broker.lastIndexOf('/') + 1);
+ return broker;
+ }
+
+ public boolean isChannelOwner() {
+ return isTargetBroker(getChannelOwner());
+ }
+
+ public CompletableFuture<String> getOwnerAsync(String serviceUnit) {
+ validateChannelState(Started, true);
+ ServiceUnitStateData data = tableview.get(serviceUnit);
+ if (data == null) {
+ return null;
+ }
+ switch (data.state()) {
+ case Owned, Splitting -> {
+ return CompletableFuture.completedFuture(data.broker());
+ }
+ case Assigned, Released -> {
+ return deferGetOwnerRequest(serviceUnit);
+ }
+ default -> {
+ return null;
+ }
+ }
+ }
+
+ public CompletableFuture<String> publishAssignEventAsync(String serviceUnit, String broker) {
+ CompletableFuture<String> getOwnerRequest = deferGetOwnerRequest(serviceUnit);
+ pubAsync(serviceUnit, new ServiceUnitStateData(Assigned, broker))
+ .whenComplete((__, ex) -> {
+ if (ex != null) {
+ getOwnerRequests.remove(serviceUnit);
Review Comment:
updated.
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,778 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Free;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Closed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Constructed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.LeaderElectionServiceStarted;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Started;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private final Schema<ServiceUnitStateData> schema;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private final String lookupServiceAddress;
+ // TODO: define BrokerRegistry
+ private final Semaphore outstandingCleanupTombstoneMessages;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+ private final LeaderElectionService leaderElectionService;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private ScheduledFuture<?> cleanupTasks;
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+ private volatile ChannelState channelState;
+
+ enum ChannelState {
+ Closed(0),
+ Constructed(1),
+ LeaderElectionServiceStarted(2),
+ Started(3);
+
+ ChannelState(int id) {
+ this.id = id;
+ }
+ int id;
+ }
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ this.lookupServiceAddress = pulsar.getLookupServiceAddress();
+ this.schema = Schema.JSON(ServiceUnitStateData.class);
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder().build();
+ this.cleanupJobs = ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder().build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} is the leader now.", lookupServiceAddress);
+ // TODO: schedule monitorOwnerships by brokerRegistry
+ } else {
+ log.debug("This broker:{} is a follower now.", lookupServiceAddress);
+ // TODO: cancel scheduled monitorOwnerships if any
+ }
+ });
+ this.channelState = Constructed;
+ }
+
+ public synchronized void start() throws PulsarServerException {
+ validateChannelState(LeaderElectionServiceStarted, false);
+ try {
+ leaderElectionService.start();
+ this.channelState = LeaderElectionServiceStarted;
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(schema)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(schema)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ channelState = Started;
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public synchronized void close() throws PulsarServerException {
+ channelState = Closed;
+ try {
+ leaderElectionService.close();
+ log.debug("Successfully closed the channel leader election service.");
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ private void validateChannelState(ChannelState targetState, boolean checkLowerIds) {
+ int order = checkLowerIds ? -1 : 1;
+ if (Integer.compare(channelState.id, targetState.id) * order > 0) {
+ throw new IllegalStateException("Invalid channel state:" + channelState.name());
+ }
+ }
+
+ public synchronized void scheduleCompaction() throws PulsarServerException {
+ try {
+ Long threshold = pulsar.getAdminClient().topicPolicies()
+ .getCompactionThreshold(TOPIC);
+ if (threshold == null || threshold == 0) {
+ pulsar.getAdminClient().topicPolicies()
+ .setCompactionThreshold(TOPIC, COMPACTION_THRESHOLD);
+ log.info("Scheduled compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ } else {
+ log.info("Already set compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ }
+ } catch (PulsarAdminException e) {
+ throw new PulsarServerException("Failed to schedule compaction.", e);
+ }
+ }
+
+ public String getChannelOwner() {
+ validateChannelState(LeaderElectionServiceStarted, true);
+ CompletableFuture<Optional<LeaderBroker>> future = leaderElectionService.readCurrentLeader();
+ if (!future.isDone()) {
+ return null;
+ }
+ Optional<LeaderBroker> leader = future.join();
+ if (leader.isEmpty()) {
+ return null;
+ }
+ //expecting http://broker-xyz:port
+ // TODO: discard this protocol prefix removal by a util func that returns lookupServiceAddress(serviceUrl)
+ String broker = leader.get().getServiceUrl();
+ broker = broker.substring(broker.lastIndexOf('/') + 1);
+ return broker;
+ }
+
+ public boolean isChannelOwner() {
+ return isTargetBroker(getChannelOwner());
+ }
+
+ public CompletableFuture<String> getOwnerAsync(String serviceUnit) {
+ validateChannelState(Started, true);
+ ServiceUnitStateData data = tableview.get(serviceUnit);
+ if (data == null) {
+ return null;
+ }
+ switch (data.state()) {
+ case Owned, Splitting -> {
+ return CompletableFuture.completedFuture(data.broker());
+ }
+ case Assigned, Released -> {
+ return deferGetOwnerRequest(serviceUnit);
+ }
+ default -> {
+ return null;
+ }
+ }
+ }
+
+ public CompletableFuture<String> publishAssignEventAsync(String serviceUnit, String broker) {
+ CompletableFuture<String> getOwnerRequest = deferGetOwnerRequest(serviceUnit);
+ pubAsync(serviceUnit, new ServiceUnitStateData(Assigned, broker))
+ .whenComplete((__, ex) -> {
+ if (ex != null) {
+ getOwnerRequests.remove(serviceUnit);
+ if (!getOwnerRequest.isCompletedExceptionally()) {
+ getOwnerRequest.completeExceptionally(ex);
+ }
+ }
+ });
+
+ return getOwnerRequest;
+ }
+
+ public CompletableFuture<Void> publishUnloadEventAsync(Unload unload) {
+ String serviceUnit = unload.serviceUnit();
+ if (isTransferCommand(unload)) {
+ ServiceUnitStateData next = new ServiceUnitStateData(Assigned,
+ unload.destBroker().get(), unload.sourceBroker());
+ return pubAsync(serviceUnit, next).thenAccept(__ -> {
+ });
+ }
+ return tombstoneAsync(serviceUnit).thenAccept(__ -> {
+ });
+ }
+
+ public CompletableFuture<Void> publishSplitEventAsync(Split split) {
+ String serviceUnit = split.serviceUnit();
+ ServiceUnitStateData next = new ServiceUnitStateData(Splitting, split.sourceBroker());
+ return pubAsync(serviceUnit, next).thenAccept(__ -> {
+ });
+ }
+
+ private void handle(String serviceUnit, ServiceUnitStateData data) {
+ if (log.isDebugEnabled()) {
+ log.info("{} received a handle request for serviceUnit:{}, data:{}",
+ lookupServiceAddress, serviceUnit, data);
+ }
+
+ ServiceUnitState state = data == null ? Free : data.state();
+
+ // TODO : Add state validation in tableview by the compaction strategy
+ switch (state) {
+ case Owned -> handleOwnEvent(serviceUnit, data);
+ case Assigned -> handleAssignEvent(serviceUnit, data);
+ case Released -> handleReleaseEvent(serviceUnit, data);
+ case Splitting -> handleSplitEvent(serviceUnit, data);
+ case Free -> handleFreeEvent(serviceUnit);
+ default -> throw new IllegalStateException("Failed to handle channel data:" + data);
+ }
+ }
+
+ private static boolean isTransferCommand(ServiceUnitStateData data) {
+ if (data == null) {
+ return false;
+ }
+ return StringUtils.isNotEmpty(data.sourceBroker());
+ }
+
+ private static boolean isTransferCommand(Unload data) {
+ return data.destBroker().isPresent();
+ }
+
+ private static String getLogEventTag(ServiceUnitStateData data) {
+ return data == null ? "Free" :
+ isTransferCommand(data) ? "Transfer:" + data.state() : data.state().toString();
+ }
+
+ private void log(Throwable e, String serviceUnit, ServiceUnitStateData data, ServiceUnitStateData next) {
+ if (e == null) {
+ if (log.isDebugEnabled() || isTransferCommand(data)) {
+ log.info("{} handled {} event for serviceUnit:{}, cur:{}, next:{}",
+ lookupServiceAddress, getLogEventTag(data), serviceUnit,
+ data == null ? "" : data,
+ next == null ? "" : next);
+ }
+ } else {
+ log.error("{} failed to handle {} event for serviceUnit:{}, cur:{}, next:{}",
+ lookupServiceAddress, getLogEventTag(data), serviceUnit,
+ data == null ? "" : data,
+ next == null ? "" : next,
+ e);
+ }
+ }
+
+ private void handleOwnEvent(String serviceUnit, ServiceUnitStateData data) {
+ var getOwnerRequest = getOwnerRequests.remove(serviceUnit);
+ if (getOwnerRequest != null) {
+ getOwnerRequest.complete(data.broker());
+ }
+ if (isTargetBroker(data.broker())) {
+ log(null, serviceUnit, data, null);
+ }
+
+ }
+
+ private void handleAssignEvent(String serviceUnit, ServiceUnitStateData data) {
+ deferGetOwnerRequest(serviceUnit);
+ if (isTargetBroker(data.broker())) {
+ ServiceUnitStateData next = new ServiceUnitStateData(
+ isTransferCommand(data) ? Released : Owned, data.broker(), data.sourceBroker());
+ pubAsync(serviceUnit, next)
+ .whenComplete((__, e) -> log(e, serviceUnit, data, next));
+ }
+ }
+
+ private void handleReleaseEvent(String serviceUnit, ServiceUnitStateData data) {
+
+ if (isTargetBroker(data.sourceBroker())) {
+ ServiceUnitStateData next = new ServiceUnitStateData(Owned, data.broker(), data.sourceBroker());
+ // TODO: when close, pass message to clients to connect to the new broker
+ closeServiceUnit(serviceUnit)
+ .thenCompose(__ -> pubAsync(serviceUnit, next))
+ .whenComplete((__, e) -> log(e, serviceUnit, data, next));
+ }
+ }
+
+ private void handleSplitEvent(String serviceUnit, ServiceUnitStateData data) {
+ if (isTargetBroker(data.broker())) {
+ splitServiceUnit(serviceUnit)
+ .thenCompose(__ -> tombstoneAsync(serviceUnit))
+ .whenComplete((__, e) -> log(e, serviceUnit, data, null));
+ }
+ }
+
+ private void handleFreeEvent(String serviceUnit) {
+ closeServiceUnit(serviceUnit)
+ .thenAccept(__ -> {
+ var request = getOwnerRequests.remove(serviceUnit);
+ if (request != null) {
+ request.completeExceptionally(new IllegalStateException("The ownership has been unloaded. "
+ + "No owner is found for serviceUnit: " + serviceUnit));
+ }
+ })
+ .whenComplete((__, e) -> log(e, serviceUnit, null, null));
+ }
+
+ private CompletableFuture<MessageId> pubAsync(String serviceUnit, ServiceUnitStateData data) {
+ validateChannelState(Started, true);
+ CompletableFuture<MessageId> future = new CompletableFuture<>();
+ producer.newMessage()
+ .key(serviceUnit)
+ .value(data)
+ .sendAsync()
+ .whenComplete((messageId, e) -> {
+ if (e != null) {
+ log.error("Failed to publish the message: serviceUnit:{}, data:{}",
+ serviceUnit, data, e);
+ future.completeExceptionally(e);
+ } else {
+ future.complete(messageId);
+ }
+ });
+ return future;
+ }
+
+ private CompletableFuture<MessageId> tombstoneAsync(String serviceUnit) {
+ return pubAsync(serviceUnit, null);
+ }
+
+ private boolean isTargetBroker(String broker) {
+ if (broker == null) {
+ return false;
+ }
+ return broker.equals(lookupServiceAddress);
+ }
+
+ private NamespaceBundle getNamespaceBundle(String bundle) {
+ final String namespaceName = LoadManagerShared.getNamespaceNameFromBundleName(bundle);
+ final String bundleRange = LoadManagerShared.getBundleRangeFromBundleName(bundle);
+ return pulsar.getNamespaceService().getNamespaceBundleFactory().getBundle(namespaceName, bundleRange);
+ }
+
+ private CompletableFuture<String> deferGetOwnerRequest(String serviceUnit) {
+ return getOwnerRequests
+ .computeIfAbsent(serviceUnit, k -> {
+ CompletableFuture future = new CompletableFuture<>();
+ future.orTimeout(inFlightStateWaitingTimeInMillis, TimeUnit.MILLISECONDS)
+ .whenComplete((v, e) -> {
+ if (e != null) {
+ getOwnerRequests.remove(serviceUnit);
Review Comment:
updated.
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,778 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Free;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Closed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Constructed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.LeaderElectionServiceStarted;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Started;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private final Schema<ServiceUnitStateData> schema;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private final String lookupServiceAddress;
+ // TODO: define BrokerRegistry
+ private final Semaphore outstandingCleanupTombstoneMessages;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+ private final LeaderElectionService leaderElectionService;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private ScheduledFuture<?> cleanupTasks;
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+ private volatile ChannelState channelState;
+
+ enum ChannelState {
+ Closed(0),
+ Constructed(1),
+ LeaderElectionServiceStarted(2),
+ Started(3);
+
+ ChannelState(int id) {
+ this.id = id;
+ }
+ int id;
+ }
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ this.lookupServiceAddress = pulsar.getLookupServiceAddress();
+ this.schema = Schema.JSON(ServiceUnitStateData.class);
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder().build();
+ this.cleanupJobs = ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder().build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} is the leader now.", lookupServiceAddress);
+ // TODO: schedule monitorOwnerships by brokerRegistry
+ } else {
+ log.debug("This broker:{} is a follower now.", lookupServiceAddress);
+ // TODO: cancel scheduled monitorOwnerships if any
+ }
+ });
+ this.channelState = Constructed;
+ }
+
+ public synchronized void start() throws PulsarServerException {
+ validateChannelState(LeaderElectionServiceStarted, false);
+ try {
+ leaderElectionService.start();
+ this.channelState = LeaderElectionServiceStarted;
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(schema)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(schema)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ channelState = Started;
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public synchronized void close() throws PulsarServerException {
+ channelState = Closed;
+ try {
+ leaderElectionService.close();
+ log.debug("Successfully closed the channel leader election service.");
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ private void validateChannelState(ChannelState targetState, boolean checkLowerIds) {
+ int order = checkLowerIds ? -1 : 1;
+ if (Integer.compare(channelState.id, targetState.id) * order > 0) {
+ throw new IllegalStateException("Invalid channel state:" + channelState.name());
+ }
+ }
+
+ public synchronized void scheduleCompaction() throws PulsarServerException {
+ try {
+ Long threshold = pulsar.getAdminClient().topicPolicies()
+ .getCompactionThreshold(TOPIC);
+ if (threshold == null || threshold == 0) {
+ pulsar.getAdminClient().topicPolicies()
+ .setCompactionThreshold(TOPIC, COMPACTION_THRESHOLD);
+ log.info("Scheduled compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ } else {
+ log.info("Already set compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ }
+ } catch (PulsarAdminException e) {
+ throw new PulsarServerException("Failed to schedule compaction.", e);
+ }
+ }
+
+ public String getChannelOwner() {
+ validateChannelState(LeaderElectionServiceStarted, true);
+ CompletableFuture<Optional<LeaderBroker>> future = leaderElectionService.readCurrentLeader();
+ if (!future.isDone()) {
+ return null;
+ }
+ Optional<LeaderBroker> leader = future.join();
+ if (leader.isEmpty()) {
+ return null;
+ }
+ //expecting http://broker-xyz:port
+ // TODO: discard this protocol prefix removal by a util func that returns lookupServiceAddress(serviceUrl)
+ String broker = leader.get().getServiceUrl();
+ broker = broker.substring(broker.lastIndexOf('/') + 1);
+ return broker;
+ }
+
+ public boolean isChannelOwner() {
+ return isTargetBroker(getChannelOwner());
+ }
+
+ public CompletableFuture<String> getOwnerAsync(String serviceUnit) {
+ validateChannelState(Started, true);
+ ServiceUnitStateData data = tableview.get(serviceUnit);
+ if (data == null) {
+ return null;
+ }
+ switch (data.state()) {
+ case Owned, Splitting -> {
+ return CompletableFuture.completedFuture(data.broker());
+ }
+ case Assigned, Released -> {
+ return deferGetOwnerRequest(serviceUnit);
+ }
+ default -> {
+ return null;
+ }
+ }
+ }
+
+ public CompletableFuture<String> publishAssignEventAsync(String serviceUnit, String broker) {
+ CompletableFuture<String> getOwnerRequest = deferGetOwnerRequest(serviceUnit);
+ pubAsync(serviceUnit, new ServiceUnitStateData(Assigned, broker))
+ .whenComplete((__, ex) -> {
+ if (ex != null) {
+ getOwnerRequests.remove(serviceUnit);
+ if (!getOwnerRequest.isCompletedExceptionally()) {
+ getOwnerRequest.completeExceptionally(ex);
+ }
+ }
+ });
+
+ return getOwnerRequest;
+ }
+
+ public CompletableFuture<Void> publishUnloadEventAsync(Unload unload) {
+ String serviceUnit = unload.serviceUnit();
+ if (isTransferCommand(unload)) {
+ ServiceUnitStateData next = new ServiceUnitStateData(Assigned,
+ unload.destBroker().get(), unload.sourceBroker());
+ return pubAsync(serviceUnit, next).thenAccept(__ -> {
+ });
+ }
+ return tombstoneAsync(serviceUnit).thenAccept(__ -> {
+ });
+ }
+
+ public CompletableFuture<Void> publishSplitEventAsync(Split split) {
+ String serviceUnit = split.serviceUnit();
+ ServiceUnitStateData next = new ServiceUnitStateData(Splitting, split.sourceBroker());
+ return pubAsync(serviceUnit, next).thenAccept(__ -> {
+ });
+ }
+
+ private void handle(String serviceUnit, ServiceUnitStateData data) {
+ if (log.isDebugEnabled()) {
+ log.info("{} received a handle request for serviceUnit:{}, data:{}",
+ lookupServiceAddress, serviceUnit, data);
+ }
+
+ ServiceUnitState state = data == null ? Free : data.state();
+
+ // TODO : Add state validation in tableview by the compaction strategy
+ switch (state) {
+ case Owned -> handleOwnEvent(serviceUnit, data);
+ case Assigned -> handleAssignEvent(serviceUnit, data);
+ case Released -> handleReleaseEvent(serviceUnit, data);
+ case Splitting -> handleSplitEvent(serviceUnit, data);
+ case Free -> handleFreeEvent(serviceUnit);
+ default -> throw new IllegalStateException("Failed to handle channel data:" + data);
+ }
+ }
+
+ private static boolean isTransferCommand(ServiceUnitStateData data) {
+ if (data == null) {
+ return false;
+ }
+ return StringUtils.isNotEmpty(data.sourceBroker());
+ }
+
+ private static boolean isTransferCommand(Unload data) {
+ return data.destBroker().isPresent();
+ }
+
+ private static String getLogEventTag(ServiceUnitStateData data) {
+ return data == null ? "Free" :
+ isTransferCommand(data) ? "Transfer:" + data.state() : data.state().toString();
+ }
+
+ private void log(Throwable e, String serviceUnit, ServiceUnitStateData data, ServiceUnitStateData next) {
+ if (e == null) {
+ if (log.isDebugEnabled() || isTransferCommand(data)) {
+ log.info("{} handled {} event for serviceUnit:{}, cur:{}, next:{}",
+ lookupServiceAddress, getLogEventTag(data), serviceUnit,
+ data == null ? "" : data,
+ next == null ? "" : next);
+ }
+ } else {
+ log.error("{} failed to handle {} event for serviceUnit:{}, cur:{}, next:{}",
+ lookupServiceAddress, getLogEventTag(data), serviceUnit,
+ data == null ? "" : data,
+ next == null ? "" : next,
+ e);
+ }
+ }
+
+ private void handleOwnEvent(String serviceUnit, ServiceUnitStateData data) {
+ var getOwnerRequest = getOwnerRequests.remove(serviceUnit);
+ if (getOwnerRequest != null) {
+ getOwnerRequest.complete(data.broker());
+ }
+ if (isTargetBroker(data.broker())) {
+ log(null, serviceUnit, data, null);
+ }
+
+ }
+
+ private void handleAssignEvent(String serviceUnit, ServiceUnitStateData data) {
+ deferGetOwnerRequest(serviceUnit);
+ if (isTargetBroker(data.broker())) {
+ ServiceUnitStateData next = new ServiceUnitStateData(
+ isTransferCommand(data) ? Released : Owned, data.broker(), data.sourceBroker());
+ pubAsync(serviceUnit, next)
+ .whenComplete((__, e) -> log(e, serviceUnit, data, next));
+ }
+ }
+
+ private void handleReleaseEvent(String serviceUnit, ServiceUnitStateData data) {
+
+ if (isTargetBroker(data.sourceBroker())) {
+ ServiceUnitStateData next = new ServiceUnitStateData(Owned, data.broker(), data.sourceBroker());
+ // TODO: when close, pass message to clients to connect to the new broker
+ closeServiceUnit(serviceUnit)
+ .thenCompose(__ -> pubAsync(serviceUnit, next))
+ .whenComplete((__, e) -> log(e, serviceUnit, data, next));
+ }
+ }
+
+ private void handleSplitEvent(String serviceUnit, ServiceUnitStateData data) {
+ if (isTargetBroker(data.broker())) {
+ splitServiceUnit(serviceUnit)
+ .thenCompose(__ -> tombstoneAsync(serviceUnit))
+ .whenComplete((__, e) -> log(e, serviceUnit, data, null));
+ }
+ }
+
+ private void handleFreeEvent(String serviceUnit) {
+ closeServiceUnit(serviceUnit)
+ .thenAccept(__ -> {
+ var request = getOwnerRequests.remove(serviceUnit);
+ if (request != null) {
+ request.completeExceptionally(new IllegalStateException("The ownership has been unloaded. "
+ + "No owner is found for serviceUnit: " + serviceUnit));
+ }
+ })
+ .whenComplete((__, e) -> log(e, serviceUnit, null, null));
+ }
+
+ private CompletableFuture<MessageId> pubAsync(String serviceUnit, ServiceUnitStateData data) {
+ validateChannelState(Started, true);
+ CompletableFuture<MessageId> future = new CompletableFuture<>();
+ producer.newMessage()
+ .key(serviceUnit)
+ .value(data)
+ .sendAsync()
+ .whenComplete((messageId, e) -> {
+ if (e != null) {
+ log.error("Failed to publish the message: serviceUnit:{}, data:{}",
+ serviceUnit, data, e);
+ future.completeExceptionally(e);
+ } else {
+ future.complete(messageId);
+ }
+ });
+ return future;
+ }
+
+ private CompletableFuture<MessageId> tombstoneAsync(String serviceUnit) {
+ return pubAsync(serviceUnit, null);
+ }
+
+ private boolean isTargetBroker(String broker) {
+ if (broker == null) {
+ return false;
+ }
+ return broker.equals(lookupServiceAddress);
+ }
+
+ private NamespaceBundle getNamespaceBundle(String bundle) {
+ final String namespaceName = LoadManagerShared.getNamespaceNameFromBundleName(bundle);
+ final String bundleRange = LoadManagerShared.getBundleRangeFromBundleName(bundle);
+ return pulsar.getNamespaceService().getNamespaceBundleFactory().getBundle(namespaceName, bundleRange);
+ }
+
+ private CompletableFuture<String> deferGetOwnerRequest(String serviceUnit) {
+ return getOwnerRequests
+ .computeIfAbsent(serviceUnit, k -> {
+ CompletableFuture future = new CompletableFuture<>();
+ future.orTimeout(inFlightStateWaitingTimeInMillis, TimeUnit.MILLISECONDS)
+ .whenComplete((v, e) -> {
+ if (e != null) {
+ getOwnerRequests.remove(serviceUnit);
+ log.warn("Failed to getOwner for serviceUnit:{}",
+ serviceUnit, e);
+ future.completeExceptionally((Throwable) e);
+ }
+ }
+ );
+ return future;
+ });
+ }
+
+ private CompletableFuture<Integer> closeServiceUnit(String serviceUnit) {
+ long startTime = System.nanoTime();
+ MutableInt unloadedTopics = new MutableInt();
+ NamespaceBundle bundle = getNamespaceBundle(serviceUnit);
+ return pulsar.getBrokerService().unloadServiceUnit(
+ bundle,
+ false,
+ pulsar.getConfig().getNamespaceBundleUnloadingTimeoutMs(),
+ TimeUnit.MILLISECONDS)
+ .handle((numUnloadedTopics, ex) -> {
+ if (ex != null) {
+ // ignore topic-close failure to unload bundle
+ log.error("Failed to close topics under bundle:{}",
+ bundle.toString(), ex);
+ } else {
+ unloadedTopics.setValue(numUnloadedTopics);
+ }
+ // clean up topics that failed to unload from the broker ownership cache
+ pulsar.getBrokerService().cleanUnloadedTopicFromCache(bundle);
+ return numUnloadedTopics;
+ })
+ .whenComplete((ignored, ex) -> {
+ double unloadBundleTime = TimeUnit.NANOSECONDS
+ .toMillis((System.nanoTime() - startTime));
+ log.info("Unloading bundle:{} with {} topics completed in {} ms",
+ bundle, unloadedTopics, unloadBundleTime, ex);
+ });
+ }
+
+ private CompletableFuture<Void> splitServiceUnit(String serviceUnit) {
+ // TODO: after the split we need to write the child ownerships to BSC instead of ZK.
+ // Use the configured NamespaceBundleSplitAlgorithm
+ long startTime = System.nanoTime();
+ return pulsar.getNamespaceService()
+ .splitAndOwnBundle(getNamespaceBundle(serviceUnit),
+ false,
+ NamespaceBundleSplitAlgorithm.RANGE_EQUALLY_DIVIDE_ALGO,
Review Comment:
updated.
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,778 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Free;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Closed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Constructed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.LeaderElectionServiceStarted;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Started;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private final Schema<ServiceUnitStateData> schema;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private final String lookupServiceAddress;
+ // TODO: define BrokerRegistry
+ private final Semaphore outstandingCleanupTombstoneMessages;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+ private final LeaderElectionService leaderElectionService;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private ScheduledFuture<?> cleanupTasks;
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+ private volatile ChannelState channelState;
+
+ enum ChannelState {
+ Closed(0),
+ Constructed(1),
+ LeaderElectionServiceStarted(2),
+ Started(3);
+
+ ChannelState(int id) {
+ this.id = id;
+ }
+ int id;
+ }
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ this.lookupServiceAddress = pulsar.getLookupServiceAddress();
+ this.schema = Schema.JSON(ServiceUnitStateData.class);
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder().build();
+ this.cleanupJobs = ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder().build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} is the leader now.", lookupServiceAddress);
+ // TODO: schedule monitorOwnerships by brokerRegistry
+ } else {
+ log.debug("This broker:{} is a follower now.", lookupServiceAddress);
+ // TODO: cancel scheduled monitorOwnerships if any
+ }
+ });
+ this.channelState = Constructed;
+ }
+
+ public synchronized void start() throws PulsarServerException {
+ validateChannelState(LeaderElectionServiceStarted, false);
+ try {
+ leaderElectionService.start();
+ this.channelState = LeaderElectionServiceStarted;
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(schema)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(schema)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ channelState = Started;
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public synchronized void close() throws PulsarServerException {
+ channelState = Closed;
+ try {
+ leaderElectionService.close();
+ log.debug("Successfully closed the channel leader election service.");
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ private void validateChannelState(ChannelState targetState, boolean checkLowerIds) {
+ int order = checkLowerIds ? -1 : 1;
+ if (Integer.compare(channelState.id, targetState.id) * order > 0) {
+ throw new IllegalStateException("Invalid channel state:" + channelState.name());
+ }
+ }
+
+ public synchronized void scheduleCompaction() throws PulsarServerException {
+ try {
+ Long threshold = pulsar.getAdminClient().topicPolicies()
+ .getCompactionThreshold(TOPIC);
+ if (threshold == null || threshold == 0) {
+ pulsar.getAdminClient().topicPolicies()
+ .setCompactionThreshold(TOPIC, COMPACTION_THRESHOLD);
+ log.info("Scheduled compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ } else {
+ log.info("Already set compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ }
+ } catch (PulsarAdminException e) {
+ throw new PulsarServerException("Failed to schedule compaction.", e);
+ }
+ }
+
+ public String getChannelOwner() {
+ validateChannelState(LeaderElectionServiceStarted, true);
+ CompletableFuture<Optional<LeaderBroker>> future = leaderElectionService.readCurrentLeader();
+ if (!future.isDone()) {
+ return null;
+ }
+ Optional<LeaderBroker> leader = future.join();
+ if (leader.isEmpty()) {
+ return null;
+ }
+ //expecting http://broker-xyz:port
+ // TODO: discard this protocol prefix removal by a util func that returns lookupServiceAddress(serviceUrl)
+ String broker = leader.get().getServiceUrl();
+ broker = broker.substring(broker.lastIndexOf('/') + 1);
+ return broker;
+ }
+
+ public boolean isChannelOwner() {
+ return isTargetBroker(getChannelOwner());
+ }
+
+ public CompletableFuture<String> getOwnerAsync(String serviceUnit) {
+ validateChannelState(Started, true);
+ ServiceUnitStateData data = tableview.get(serviceUnit);
+ if (data == null) {
+ return null;
+ }
+ switch (data.state()) {
+ case Owned, Splitting -> {
+ return CompletableFuture.completedFuture(data.broker());
+ }
+ case Assigned, Released -> {
+ return deferGetOwnerRequest(serviceUnit);
+ }
+ default -> {
+ return null;
+ }
+ }
+ }
+
+ public CompletableFuture<String> publishAssignEventAsync(String serviceUnit, String broker) {
+ CompletableFuture<String> getOwnerRequest = deferGetOwnerRequest(serviceUnit);
+ pubAsync(serviceUnit, new ServiceUnitStateData(Assigned, broker))
+ .whenComplete((__, ex) -> {
+ if (ex != null) {
+ getOwnerRequests.remove(serviceUnit);
+ if (!getOwnerRequest.isCompletedExceptionally()) {
+ getOwnerRequest.completeExceptionally(ex);
+ }
+ }
+ });
+
+ return getOwnerRequest;
+ }
+
+ public CompletableFuture<Void> publishUnloadEventAsync(Unload unload) {
+ String serviceUnit = unload.serviceUnit();
+ if (isTransferCommand(unload)) {
+ ServiceUnitStateData next = new ServiceUnitStateData(Assigned,
+ unload.destBroker().get(), unload.sourceBroker());
+ return pubAsync(serviceUnit, next).thenAccept(__ -> {
+ });
+ }
+ return tombstoneAsync(serviceUnit).thenAccept(__ -> {
+ });
+ }
+
+ public CompletableFuture<Void> publishSplitEventAsync(Split split) {
+ String serviceUnit = split.serviceUnit();
+ ServiceUnitStateData next = new ServiceUnitStateData(Splitting, split.sourceBroker());
+ return pubAsync(serviceUnit, next).thenAccept(__ -> {
+ });
+ }
+
+ private void handle(String serviceUnit, ServiceUnitStateData data) {
+ if (log.isDebugEnabled()) {
+ log.info("{} received a handle request for serviceUnit:{}, data:{}",
+ lookupServiceAddress, serviceUnit, data);
+ }
+
+ ServiceUnitState state = data == null ? Free : data.state();
+
+ // TODO : Add state validation in tableview by the compaction strategy
+ switch (state) {
+ case Owned -> handleOwnEvent(serviceUnit, data);
+ case Assigned -> handleAssignEvent(serviceUnit, data);
+ case Released -> handleReleaseEvent(serviceUnit, data);
+ case Splitting -> handleSplitEvent(serviceUnit, data);
+ case Free -> handleFreeEvent(serviceUnit);
+ default -> throw new IllegalStateException("Failed to handle channel data:" + data);
+ }
+ }
+
+ private static boolean isTransferCommand(ServiceUnitStateData data) {
+ if (data == null) {
+ return false;
+ }
+ return StringUtils.isNotEmpty(data.sourceBroker());
+ }
+
+ private static boolean isTransferCommand(Unload data) {
+ return data.destBroker().isPresent();
+ }
+
+ private static String getLogEventTag(ServiceUnitStateData data) {
+ return data == null ? "Free" :
+ isTransferCommand(data) ? "Transfer:" + data.state() : data.state().toString();
+ }
+
+ private void log(Throwable e, String serviceUnit, ServiceUnitStateData data, ServiceUnitStateData next) {
+ if (e == null) {
+ if (log.isDebugEnabled() || isTransferCommand(data)) {
+ log.info("{} handled {} event for serviceUnit:{}, cur:{}, next:{}",
+ lookupServiceAddress, getLogEventTag(data), serviceUnit,
+ data == null ? "" : data,
+ next == null ? "" : next);
+ }
+ } else {
+ log.error("{} failed to handle {} event for serviceUnit:{}, cur:{}, next:{}",
+ lookupServiceAddress, getLogEventTag(data), serviceUnit,
+ data == null ? "" : data,
+ next == null ? "" : next,
+ e);
+ }
+ }
+
+ private void handleOwnEvent(String serviceUnit, ServiceUnitStateData data) {
+ var getOwnerRequest = getOwnerRequests.remove(serviceUnit);
+ if (getOwnerRequest != null) {
+ getOwnerRequest.complete(data.broker());
+ }
+ if (isTargetBroker(data.broker())) {
+ log(null, serviceUnit, data, null);
+ }
+
+ }
+
+ private void handleAssignEvent(String serviceUnit, ServiceUnitStateData data) {
+ deferGetOwnerRequest(serviceUnit);
+ if (isTargetBroker(data.broker())) {
+ ServiceUnitStateData next = new ServiceUnitStateData(
+ isTransferCommand(data) ? Released : Owned, data.broker(), data.sourceBroker());
+ pubAsync(serviceUnit, next)
+ .whenComplete((__, e) -> log(e, serviceUnit, data, next));
+ }
+ }
+
+ private void handleReleaseEvent(String serviceUnit, ServiceUnitStateData data) {
+
+ if (isTargetBroker(data.sourceBroker())) {
+ ServiceUnitStateData next = new ServiceUnitStateData(Owned, data.broker(), data.sourceBroker());
+ // TODO: when close, pass message to clients to connect to the new broker
+ closeServiceUnit(serviceUnit)
+ .thenCompose(__ -> pubAsync(serviceUnit, next))
+ .whenComplete((__, e) -> log(e, serviceUnit, data, next));
+ }
+ }
+
+ private void handleSplitEvent(String serviceUnit, ServiceUnitStateData data) {
+ if (isTargetBroker(data.broker())) {
+ splitServiceUnit(serviceUnit)
+ .thenCompose(__ -> tombstoneAsync(serviceUnit))
+ .whenComplete((__, e) -> log(e, serviceUnit, data, null));
+ }
+ }
+
+ private void handleFreeEvent(String serviceUnit) {
+ closeServiceUnit(serviceUnit)
+ .thenAccept(__ -> {
+ var request = getOwnerRequests.remove(serviceUnit);
+ if (request != null) {
+ request.completeExceptionally(new IllegalStateException("The ownership has been unloaded. "
+ + "No owner is found for serviceUnit: " + serviceUnit));
+ }
+ })
+ .whenComplete((__, e) -> log(e, serviceUnit, null, null));
+ }
+
+ private CompletableFuture<MessageId> pubAsync(String serviceUnit, ServiceUnitStateData data) {
+ validateChannelState(Started, true);
+ CompletableFuture<MessageId> future = new CompletableFuture<>();
+ producer.newMessage()
+ .key(serviceUnit)
+ .value(data)
+ .sendAsync()
+ .whenComplete((messageId, e) -> {
+ if (e != null) {
+ log.error("Failed to publish the message: serviceUnit:{}, data:{}",
+ serviceUnit, data, e);
+ future.completeExceptionally(e);
+ } else {
+ future.complete(messageId);
+ }
+ });
+ return future;
+ }
+
+ private CompletableFuture<MessageId> tombstoneAsync(String serviceUnit) {
+ return pubAsync(serviceUnit, null);
+ }
+
+ private boolean isTargetBroker(String broker) {
+ if (broker == null) {
+ return false;
+ }
+ return broker.equals(lookupServiceAddress);
+ }
+
+ private NamespaceBundle getNamespaceBundle(String bundle) {
+ final String namespaceName = LoadManagerShared.getNamespaceNameFromBundleName(bundle);
+ final String bundleRange = LoadManagerShared.getBundleRangeFromBundleName(bundle);
+ return pulsar.getNamespaceService().getNamespaceBundleFactory().getBundle(namespaceName, bundleRange);
+ }
+
+ private CompletableFuture<String> deferGetOwnerRequest(String serviceUnit) {
+ return getOwnerRequests
+ .computeIfAbsent(serviceUnit, k -> {
+ CompletableFuture future = new CompletableFuture<>();
+ future.orTimeout(inFlightStateWaitingTimeInMillis, TimeUnit.MILLISECONDS)
+ .whenComplete((v, e) -> {
+ if (e != null) {
+ getOwnerRequests.remove(serviceUnit);
+ log.warn("Failed to getOwner for serviceUnit:{}",
+ serviceUnit, e);
+ future.completeExceptionally((Throwable) e);
+ }
+ }
+ );
+ return future;
+ });
+ }
+
+ private CompletableFuture<Integer> closeServiceUnit(String serviceUnit) {
+ long startTime = System.nanoTime();
+ MutableInt unloadedTopics = new MutableInt();
+ NamespaceBundle bundle = getNamespaceBundle(serviceUnit);
+ return pulsar.getBrokerService().unloadServiceUnit(
+ bundle,
+ false,
+ pulsar.getConfig().getNamespaceBundleUnloadingTimeoutMs(),
+ TimeUnit.MILLISECONDS)
+ .handle((numUnloadedTopics, ex) -> {
+ if (ex != null) {
+ // ignore topic-close failure to unload bundle
+ log.error("Failed to close topics under bundle:{}",
+ bundle.toString(), ex);
+ } else {
+ unloadedTopics.setValue(numUnloadedTopics);
+ }
+ // clean up topics that failed to unload from the broker ownership cache
+ pulsar.getBrokerService().cleanUnloadedTopicFromCache(bundle);
+ return numUnloadedTopics;
+ })
+ .whenComplete((ignored, ex) -> {
+ double unloadBundleTime = TimeUnit.NANOSECONDS
+ .toMillis((System.nanoTime() - startTime));
+ log.info("Unloading bundle:{} with {} topics completed in {} ms",
+ bundle, unloadedTopics, unloadBundleTime, ex);
Review Comment:
This part is from the original code, but updated.
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,778 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Free;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Closed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Constructed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.LeaderElectionServiceStarted;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Started;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private final Schema<ServiceUnitStateData> schema;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private final String lookupServiceAddress;
+ // TODO: define BrokerRegistry
+ private final Semaphore outstandingCleanupTombstoneMessages;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+ private final LeaderElectionService leaderElectionService;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private ScheduledFuture<?> cleanupTasks;
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+ private volatile ChannelState channelState;
+
+ enum ChannelState {
+ Closed(0),
+ Constructed(1),
+ LeaderElectionServiceStarted(2),
+ Started(3);
+
+ ChannelState(int id) {
+ this.id = id;
+ }
+ int id;
+ }
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ this.lookupServiceAddress = pulsar.getLookupServiceAddress();
+ this.schema = Schema.JSON(ServiceUnitStateData.class);
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder().build();
+ this.cleanupJobs = ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder().build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} is the leader now.", lookupServiceAddress);
+ // TODO: schedule monitorOwnerships by brokerRegistry
+ } else {
+ log.debug("This broker:{} is a follower now.", lookupServiceAddress);
+ // TODO: cancel scheduled monitorOwnerships if any
+ }
+ });
+ this.channelState = Constructed;
+ }
+
+ public synchronized void start() throws PulsarServerException {
+ validateChannelState(LeaderElectionServiceStarted, false);
+ try {
+ leaderElectionService.start();
+ this.channelState = LeaderElectionServiceStarted;
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(schema)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(schema)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ channelState = Started;
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public synchronized void close() throws PulsarServerException {
+ channelState = Closed;
+ try {
+ leaderElectionService.close();
+ log.debug("Successfully closed the channel leader election service.");
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ private void validateChannelState(ChannelState targetState, boolean checkLowerIds) {
+ int order = checkLowerIds ? -1 : 1;
+ if (Integer.compare(channelState.id, targetState.id) * order > 0) {
+ throw new IllegalStateException("Invalid channel state:" + channelState.name());
+ }
+ }
+
+ public synchronized void scheduleCompaction() throws PulsarServerException {
+ try {
+ Long threshold = pulsar.getAdminClient().topicPolicies()
+ .getCompactionThreshold(TOPIC);
+ if (threshold == null || threshold == 0) {
+ pulsar.getAdminClient().topicPolicies()
+ .setCompactionThreshold(TOPIC, COMPACTION_THRESHOLD);
+ log.info("Scheduled compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ } else {
+ log.info("Already set compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ }
+ } catch (PulsarAdminException e) {
+ throw new PulsarServerException("Failed to schedule compaction.", e);
+ }
+ }
+
+ public String getChannelOwner() {
+ validateChannelState(LeaderElectionServiceStarted, true);
+ CompletableFuture<Optional<LeaderBroker>> future = leaderElectionService.readCurrentLeader();
+ if (!future.isDone()) {
+ return null;
+ }
+ Optional<LeaderBroker> leader = future.join();
+ if (leader.isEmpty()) {
+ return null;
+ }
+ //expecting http://broker-xyz:port
+ // TODO: discard this protocol prefix removal by a util func that returns lookupServiceAddress(serviceUrl)
+ String broker = leader.get().getServiceUrl();
+ broker = broker.substring(broker.lastIndexOf('/') + 1);
+ return broker;
+ }
+
+ public boolean isChannelOwner() {
+ return isTargetBroker(getChannelOwner());
+ }
+
+ public CompletableFuture<String> getOwnerAsync(String serviceUnit) {
+ validateChannelState(Started, true);
+ ServiceUnitStateData data = tableview.get(serviceUnit);
+ if (data == null) {
+ return null;
+ }
+ switch (data.state()) {
+ case Owned, Splitting -> {
+ return CompletableFuture.completedFuture(data.broker());
+ }
+ case Assigned, Released -> {
+ return deferGetOwnerRequest(serviceUnit);
+ }
+ default -> {
+ return null;
+ }
+ }
+ }
+
+ public CompletableFuture<String> publishAssignEventAsync(String serviceUnit, String broker) {
+ CompletableFuture<String> getOwnerRequest = deferGetOwnerRequest(serviceUnit);
+ pubAsync(serviceUnit, new ServiceUnitStateData(Assigned, broker))
+ .whenComplete((__, ex) -> {
+ if (ex != null) {
+ getOwnerRequests.remove(serviceUnit);
+ if (!getOwnerRequest.isCompletedExceptionally()) {
+ getOwnerRequest.completeExceptionally(ex);
+ }
+ }
+ });
+
+ return getOwnerRequest;
+ }
+
+ public CompletableFuture<Void> publishUnloadEventAsync(Unload unload) {
+ String serviceUnit = unload.serviceUnit();
+ if (isTransferCommand(unload)) {
+ ServiceUnitStateData next = new ServiceUnitStateData(Assigned,
+ unload.destBroker().get(), unload.sourceBroker());
+ return pubAsync(serviceUnit, next).thenAccept(__ -> {
+ });
+ }
+ return tombstoneAsync(serviceUnit).thenAccept(__ -> {
+ });
+ }
+
+ public CompletableFuture<Void> publishSplitEventAsync(Split split) {
+ String serviceUnit = split.serviceUnit();
+ ServiceUnitStateData next = new ServiceUnitStateData(Splitting, split.sourceBroker());
+ return pubAsync(serviceUnit, next).thenAccept(__ -> {
+ });
+ }
+
+ private void handle(String serviceUnit, ServiceUnitStateData data) {
+ if (log.isDebugEnabled()) {
+ log.info("{} received a handle request for serviceUnit:{}, data:{}",
+ lookupServiceAddress, serviceUnit, data);
+ }
+
+ ServiceUnitState state = data == null ? Free : data.state();
+
+ // TODO : Add state validation in tableview by the compaction strategy
+ switch (state) {
+ case Owned -> handleOwnEvent(serviceUnit, data);
+ case Assigned -> handleAssignEvent(serviceUnit, data);
+ case Released -> handleReleaseEvent(serviceUnit, data);
+ case Splitting -> handleSplitEvent(serviceUnit, data);
+ case Free -> handleFreeEvent(serviceUnit);
+ default -> throw new IllegalStateException("Failed to handle channel data:" + data);
+ }
+ }
+
+ private static boolean isTransferCommand(ServiceUnitStateData data) {
+ if (data == null) {
+ return false;
+ }
+ return StringUtils.isNotEmpty(data.sourceBroker());
+ }
+
+ private static boolean isTransferCommand(Unload data) {
+ return data.destBroker().isPresent();
+ }
+
+ private static String getLogEventTag(ServiceUnitStateData data) {
+ return data == null ? "Free" :
+ isTransferCommand(data) ? "Transfer:" + data.state() : data.state().toString();
+ }
+
+ private void log(Throwable e, String serviceUnit, ServiceUnitStateData data, ServiceUnitStateData next) {
+ if (e == null) {
+ if (log.isDebugEnabled() || isTransferCommand(data)) {
+ log.info("{} handled {} event for serviceUnit:{}, cur:{}, next:{}",
+ lookupServiceAddress, getLogEventTag(data), serviceUnit,
+ data == null ? "" : data,
+ next == null ? "" : next);
+ }
+ } else {
+ log.error("{} failed to handle {} event for serviceUnit:{}, cur:{}, next:{}",
+ lookupServiceAddress, getLogEventTag(data), serviceUnit,
+ data == null ? "" : data,
+ next == null ? "" : next,
+ e);
+ }
+ }
+
+ private void handleOwnEvent(String serviceUnit, ServiceUnitStateData data) {
+ var getOwnerRequest = getOwnerRequests.remove(serviceUnit);
+ if (getOwnerRequest != null) {
+ getOwnerRequest.complete(data.broker());
+ }
+ if (isTargetBroker(data.broker())) {
+ log(null, serviceUnit, data, null);
+ }
+
+ }
+
+ private void handleAssignEvent(String serviceUnit, ServiceUnitStateData data) {
+ deferGetOwnerRequest(serviceUnit);
+ if (isTargetBroker(data.broker())) {
+ ServiceUnitStateData next = new ServiceUnitStateData(
+ isTransferCommand(data) ? Released : Owned, data.broker(), data.sourceBroker());
+ pubAsync(serviceUnit, next)
+ .whenComplete((__, e) -> log(e, serviceUnit, data, next));
+ }
+ }
+
+ private void handleReleaseEvent(String serviceUnit, ServiceUnitStateData data) {
+
+ if (isTargetBroker(data.sourceBroker())) {
+ ServiceUnitStateData next = new ServiceUnitStateData(Owned, data.broker(), data.sourceBroker());
+ // TODO: when close, pass message to clients to connect to the new broker
+ closeServiceUnit(serviceUnit)
+ .thenCompose(__ -> pubAsync(serviceUnit, next))
+ .whenComplete((__, e) -> log(e, serviceUnit, data, next));
+ }
+ }
+
+ private void handleSplitEvent(String serviceUnit, ServiceUnitStateData data) {
+ if (isTargetBroker(data.broker())) {
+ splitServiceUnit(serviceUnit)
+ .thenCompose(__ -> tombstoneAsync(serviceUnit))
+ .whenComplete((__, e) -> log(e, serviceUnit, data, null));
+ }
+ }
+
+ private void handleFreeEvent(String serviceUnit) {
+ closeServiceUnit(serviceUnit)
+ .thenAccept(__ -> {
+ var request = getOwnerRequests.remove(serviceUnit);
+ if (request != null) {
+ request.completeExceptionally(new IllegalStateException("The ownership has been unloaded. "
+ + "No owner is found for serviceUnit: " + serviceUnit));
+ }
+ })
+ .whenComplete((__, e) -> log(e, serviceUnit, null, null));
+ }
+
+ private CompletableFuture<MessageId> pubAsync(String serviceUnit, ServiceUnitStateData data) {
+ validateChannelState(Started, true);
+ CompletableFuture<MessageId> future = new CompletableFuture<>();
+ producer.newMessage()
+ .key(serviceUnit)
+ .value(data)
+ .sendAsync()
+ .whenComplete((messageId, e) -> {
+ if (e != null) {
+ log.error("Failed to publish the message: serviceUnit:{}, data:{}",
+ serviceUnit, data, e);
+ future.completeExceptionally(e);
+ } else {
+ future.complete(messageId);
+ }
+ });
+ return future;
+ }
+
+ private CompletableFuture<MessageId> tombstoneAsync(String serviceUnit) {
+ return pubAsync(serviceUnit, null);
+ }
+
+ private boolean isTargetBroker(String broker) {
+ if (broker == null) {
+ return false;
+ }
+ return broker.equals(lookupServiceAddress);
+ }
+
+ private NamespaceBundle getNamespaceBundle(String bundle) {
+ final String namespaceName = LoadManagerShared.getNamespaceNameFromBundleName(bundle);
+ final String bundleRange = LoadManagerShared.getBundleRangeFromBundleName(bundle);
+ return pulsar.getNamespaceService().getNamespaceBundleFactory().getBundle(namespaceName, bundleRange);
+ }
+
+ private CompletableFuture<String> deferGetOwnerRequest(String serviceUnit) {
+ return getOwnerRequests
+ .computeIfAbsent(serviceUnit, k -> {
+ CompletableFuture future = new CompletableFuture<>();
+ future.orTimeout(inFlightStateWaitingTimeInMillis, TimeUnit.MILLISECONDS)
+ .whenComplete((v, e) -> {
+ if (e != null) {
+ getOwnerRequests.remove(serviceUnit);
+ log.warn("Failed to getOwner for serviceUnit:{}",
+ serviceUnit, e);
+ future.completeExceptionally((Throwable) e);
+ }
+ }
+ );
+ return future;
+ });
+ }
+
+ private CompletableFuture<Integer> closeServiceUnit(String serviceUnit) {
+ long startTime = System.nanoTime();
+ MutableInt unloadedTopics = new MutableInt();
+ NamespaceBundle bundle = getNamespaceBundle(serviceUnit);
+ return pulsar.getBrokerService().unloadServiceUnit(
+ bundle,
+ false,
+ pulsar.getConfig().getNamespaceBundleUnloadingTimeoutMs(),
+ TimeUnit.MILLISECONDS)
+ .handle((numUnloadedTopics, ex) -> {
+ if (ex != null) {
+ // ignore topic-close failure to unload bundle
+ log.error("Failed to close topics under bundle:{}",
+ bundle.toString(), ex);
Review Comment:
updated.
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,778 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Free;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Closed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Constructed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.LeaderElectionServiceStarted;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Started;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private final Schema<ServiceUnitStateData> schema;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private final String lookupServiceAddress;
+ // TODO: define BrokerRegistry
+ private final Semaphore outstandingCleanupTombstoneMessages;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+ private final LeaderElectionService leaderElectionService;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private ScheduledFuture<?> cleanupTasks;
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+ private volatile ChannelState channelState;
+
+ enum ChannelState {
+ Closed(0),
+ Constructed(1),
+ LeaderElectionServiceStarted(2),
+ Started(3);
+
+ ChannelState(int id) {
+ this.id = id;
+ }
+ int id;
+ }
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ this.lookupServiceAddress = pulsar.getLookupServiceAddress();
+ this.schema = Schema.JSON(ServiceUnitStateData.class);
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder().build();
+ this.cleanupJobs = ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder().build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} is the leader now.", lookupServiceAddress);
+ // TODO: schedule monitorOwnerships by brokerRegistry
+ } else {
+ log.debug("This broker:{} is a follower now.", lookupServiceAddress);
+ // TODO: cancel scheduled monitorOwnerships if any
+ }
+ });
+ this.channelState = Constructed;
+ }
+
+ public synchronized void start() throws PulsarServerException {
+ validateChannelState(LeaderElectionServiceStarted, false);
+ try {
+ leaderElectionService.start();
+ this.channelState = LeaderElectionServiceStarted;
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(schema)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(schema)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ channelState = Started;
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public synchronized void close() throws PulsarServerException {
+ channelState = Closed;
+ try {
+ leaderElectionService.close();
+ log.debug("Successfully closed the channel leader election service.");
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ private void validateChannelState(ChannelState targetState, boolean checkLowerIds) {
+ int order = checkLowerIds ? -1 : 1;
+ if (Integer.compare(channelState.id, targetState.id) * order > 0) {
+ throw new IllegalStateException("Invalid channel state:" + channelState.name());
+ }
+ }
+
+ public synchronized void scheduleCompaction() throws PulsarServerException {
+ try {
+ Long threshold = pulsar.getAdminClient().topicPolicies()
+ .getCompactionThreshold(TOPIC);
+ if (threshold == null || threshold == 0) {
+ pulsar.getAdminClient().topicPolicies()
+ .setCompactionThreshold(TOPIC, COMPACTION_THRESHOLD);
+ log.info("Scheduled compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ } else {
+ log.info("Already set compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ }
+ } catch (PulsarAdminException e) {
+ throw new PulsarServerException("Failed to schedule compaction.", e);
+ }
+ }
+
+ public String getChannelOwner() {
+ validateChannelState(LeaderElectionServiceStarted, true);
+ CompletableFuture<Optional<LeaderBroker>> future = leaderElectionService.readCurrentLeader();
+ if (!future.isDone()) {
+ return null;
+ }
+ Optional<LeaderBroker> leader = future.join();
+ if (leader.isEmpty()) {
+ return null;
+ }
+ //expecting http://broker-xyz:port
+ // TODO: discard this protocol prefix removal by a util func that returns lookupServiceAddress(serviceUrl)
+ String broker = leader.get().getServiceUrl();
+ broker = broker.substring(broker.lastIndexOf('/') + 1);
+ return broker;
+ }
+
+ public boolean isChannelOwner() {
+ return isTargetBroker(getChannelOwner());
+ }
+
+ public CompletableFuture<String> getOwnerAsync(String serviceUnit) {
+ validateChannelState(Started, true);
+ ServiceUnitStateData data = tableview.get(serviceUnit);
+ if (data == null) {
+ return null;
+ }
+ switch (data.state()) {
+ case Owned, Splitting -> {
+ return CompletableFuture.completedFuture(data.broker());
+ }
+ case Assigned, Released -> {
+ return deferGetOwnerRequest(serviceUnit);
+ }
+ default -> {
+ return null;
+ }
+ }
+ }
+
+ public CompletableFuture<String> publishAssignEventAsync(String serviceUnit, String broker) {
+ CompletableFuture<String> getOwnerRequest = deferGetOwnerRequest(serviceUnit);
+ pubAsync(serviceUnit, new ServiceUnitStateData(Assigned, broker))
+ .whenComplete((__, ex) -> {
+ if (ex != null) {
+ getOwnerRequests.remove(serviceUnit);
+ if (!getOwnerRequest.isCompletedExceptionally()) {
+ getOwnerRequest.completeExceptionally(ex);
+ }
+ }
+ });
+
+ return getOwnerRequest;
+ }
+
+ public CompletableFuture<Void> publishUnloadEventAsync(Unload unload) {
+ String serviceUnit = unload.serviceUnit();
+ if (isTransferCommand(unload)) {
+ ServiceUnitStateData next = new ServiceUnitStateData(Assigned,
+ unload.destBroker().get(), unload.sourceBroker());
+ return pubAsync(serviceUnit, next).thenAccept(__ -> {
+ });
+ }
+ return tombstoneAsync(serviceUnit).thenAccept(__ -> {
+ });
Review Comment:
updated.
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,778 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Free;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Closed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Constructed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.LeaderElectionServiceStarted;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Started;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private final Schema<ServiceUnitStateData> schema;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private final String lookupServiceAddress;
+ // TODO: define BrokerRegistry
+ private final Semaphore outstandingCleanupTombstoneMessages;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+ private final LeaderElectionService leaderElectionService;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private ScheduledFuture<?> cleanupTasks;
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+ private volatile ChannelState channelState;
+
+ enum ChannelState {
+ Closed(0),
+ Constructed(1),
+ LeaderElectionServiceStarted(2),
+ Started(3);
+
+ ChannelState(int id) {
+ this.id = id;
+ }
+ int id;
+ }
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ this.lookupServiceAddress = pulsar.getLookupServiceAddress();
+ this.schema = Schema.JSON(ServiceUnitStateData.class);
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder().build();
+ this.cleanupJobs = ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder().build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} is the leader now.", lookupServiceAddress);
+ // TODO: schedule monitorOwnerships by brokerRegistry
+ } else {
+ log.debug("This broker:{} is a follower now.", lookupServiceAddress);
+ // TODO: cancel scheduled monitorOwnerships if any
+ }
+ });
+ this.channelState = Constructed;
+ }
+
+ public synchronized void start() throws PulsarServerException {
+ validateChannelState(LeaderElectionServiceStarted, false);
+ try {
+ leaderElectionService.start();
+ this.channelState = LeaderElectionServiceStarted;
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(schema)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(schema)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ channelState = Started;
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public synchronized void close() throws PulsarServerException {
+ channelState = Closed;
+ try {
+ leaderElectionService.close();
+ log.debug("Successfully closed the channel leader election service.");
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ private void validateChannelState(ChannelState targetState, boolean checkLowerIds) {
+ int order = checkLowerIds ? -1 : 1;
+ if (Integer.compare(channelState.id, targetState.id) * order > 0) {
+ throw new IllegalStateException("Invalid channel state:" + channelState.name());
+ }
+ }
+
+ public synchronized void scheduleCompaction() throws PulsarServerException {
+ try {
+ Long threshold = pulsar.getAdminClient().topicPolicies()
+ .getCompactionThreshold(TOPIC);
+ if (threshold == null || threshold == 0) {
+ pulsar.getAdminClient().topicPolicies()
+ .setCompactionThreshold(TOPIC, COMPACTION_THRESHOLD);
+ log.info("Scheduled compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ } else {
+ log.info("Already set compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ }
+ } catch (PulsarAdminException e) {
+ throw new PulsarServerException("Failed to schedule compaction.", e);
+ }
+ }
+
+ public String getChannelOwner() {
+ validateChannelState(LeaderElectionServiceStarted, true);
+ CompletableFuture<Optional<LeaderBroker>> future = leaderElectionService.readCurrentLeader();
+ if (!future.isDone()) {
+ return null;
+ }
+ Optional<LeaderBroker> leader = future.join();
+ if (leader.isEmpty()) {
+ return null;
+ }
+ //expecting http://broker-xyz:port
+ // TODO: discard this protocol prefix removal by a util func that returns lookupServiceAddress(serviceUrl)
+ String broker = leader.get().getServiceUrl();
+ broker = broker.substring(broker.lastIndexOf('/') + 1);
+ return broker;
+ }
+
+ public boolean isChannelOwner() {
+ return isTargetBroker(getChannelOwner());
+ }
+
+ public CompletableFuture<String> getOwnerAsync(String serviceUnit) {
+ validateChannelState(Started, true);
+ ServiceUnitStateData data = tableview.get(serviceUnit);
+ if (data == null) {
+ return null;
+ }
+ switch (data.state()) {
+ case Owned, Splitting -> {
+ return CompletableFuture.completedFuture(data.broker());
+ }
+ case Assigned, Released -> {
+ return deferGetOwnerRequest(serviceUnit);
+ }
+ default -> {
+ return null;
+ }
+ }
+ }
+
+ public CompletableFuture<String> publishAssignEventAsync(String serviceUnit, String broker) {
+ CompletableFuture<String> getOwnerRequest = deferGetOwnerRequest(serviceUnit);
+ pubAsync(serviceUnit, new ServiceUnitStateData(Assigned, broker))
+ .whenComplete((__, ex) -> {
+ if (ex != null) {
+ getOwnerRequests.remove(serviceUnit);
+ if (!getOwnerRequest.isCompletedExceptionally()) {
+ getOwnerRequest.completeExceptionally(ex);
+ }
+ }
+ });
+
+ return getOwnerRequest;
+ }
+
+ public CompletableFuture<Void> publishUnloadEventAsync(Unload unload) {
+ String serviceUnit = unload.serviceUnit();
+ if (isTransferCommand(unload)) {
+ ServiceUnitStateData next = new ServiceUnitStateData(Assigned,
+ unload.destBroker().get(), unload.sourceBroker());
+ return pubAsync(serviceUnit, next).thenAccept(__ -> {
+ });
+ }
+ return tombstoneAsync(serviceUnit).thenAccept(__ -> {
+ });
+ }
+
+ public CompletableFuture<Void> publishSplitEventAsync(Split split) {
+ String serviceUnit = split.serviceUnit();
+ ServiceUnitStateData next = new ServiceUnitStateData(Splitting, split.sourceBroker());
+ return pubAsync(serviceUnit, next).thenAccept(__ -> {
+ });
+ }
+
+ private void handle(String serviceUnit, ServiceUnitStateData data) {
+ if (log.isDebugEnabled()) {
+ log.info("{} received a handle request for serviceUnit:{}, data:{}",
+ lookupServiceAddress, serviceUnit, data);
+ }
+
+ ServiceUnitState state = data == null ? Free : data.state();
+
+ // TODO : Add state validation in tableview by the compaction strategy
+ switch (state) {
+ case Owned -> handleOwnEvent(serviceUnit, data);
+ case Assigned -> handleAssignEvent(serviceUnit, data);
+ case Released -> handleReleaseEvent(serviceUnit, data);
+ case Splitting -> handleSplitEvent(serviceUnit, data);
+ case Free -> handleFreeEvent(serviceUnit);
+ default -> throw new IllegalStateException("Failed to handle channel data:" + data);
+ }
+ }
+
+ private static boolean isTransferCommand(ServiceUnitStateData data) {
+ if (data == null) {
+ return false;
+ }
+ return StringUtils.isNotEmpty(data.sourceBroker());
+ }
+
+ private static boolean isTransferCommand(Unload data) {
+ return data.destBroker().isPresent();
+ }
+
+ private static String getLogEventTag(ServiceUnitStateData data) {
+ return data == null ? "Free" :
+ isTransferCommand(data) ? "Transfer:" + data.state() : data.state().toString();
+ }
+
+ private void log(Throwable e, String serviceUnit, ServiceUnitStateData data, ServiceUnitStateData next) {
+ if (e == null) {
+ if (log.isDebugEnabled() || isTransferCommand(data)) {
+ log.info("{} handled {} event for serviceUnit:{}, cur:{}, next:{}",
+ lookupServiceAddress, getLogEventTag(data), serviceUnit,
+ data == null ? "" : data,
+ next == null ? "" : next);
+ }
+ } else {
+ log.error("{} failed to handle {} event for serviceUnit:{}, cur:{}, next:{}",
+ lookupServiceAddress, getLogEventTag(data), serviceUnit,
+ data == null ? "" : data,
+ next == null ? "" : next,
+ e);
+ }
+ }
+
+ private void handleOwnEvent(String serviceUnit, ServiceUnitStateData data) {
+ var getOwnerRequest = getOwnerRequests.remove(serviceUnit);
+ if (getOwnerRequest != null) {
+ getOwnerRequest.complete(data.broker());
+ }
+ if (isTargetBroker(data.broker())) {
+ log(null, serviceUnit, data, null);
+ }
+
+ }
+
+ private void handleAssignEvent(String serviceUnit, ServiceUnitStateData data) {
+ deferGetOwnerRequest(serviceUnit);
+ if (isTargetBroker(data.broker())) {
+ ServiceUnitStateData next = new ServiceUnitStateData(
+ isTransferCommand(data) ? Released : Owned, data.broker(), data.sourceBroker());
+ pubAsync(serviceUnit, next)
+ .whenComplete((__, e) -> log(e, serviceUnit, data, next));
+ }
+ }
+
+ private void handleReleaseEvent(String serviceUnit, ServiceUnitStateData data) {
+
+ if (isTargetBroker(data.sourceBroker())) {
+ ServiceUnitStateData next = new ServiceUnitStateData(Owned, data.broker(), data.sourceBroker());
+ // TODO: when close, pass message to clients to connect to the new broker
+ closeServiceUnit(serviceUnit)
+ .thenCompose(__ -> pubAsync(serviceUnit, next))
+ .whenComplete((__, e) -> log(e, serviceUnit, data, next));
+ }
+ }
+
+ private void handleSplitEvent(String serviceUnit, ServiceUnitStateData data) {
+ if (isTargetBroker(data.broker())) {
+ splitServiceUnit(serviceUnit)
+ .thenCompose(__ -> tombstoneAsync(serviceUnit))
+ .whenComplete((__, e) -> log(e, serviceUnit, data, null));
+ }
+ }
+
+ private void handleFreeEvent(String serviceUnit) {
+ closeServiceUnit(serviceUnit)
+ .thenAccept(__ -> {
+ var request = getOwnerRequests.remove(serviceUnit);
+ if (request != null) {
+ request.completeExceptionally(new IllegalStateException("The ownership has been unloaded. "
+ + "No owner is found for serviceUnit: " + serviceUnit));
+ }
+ })
+ .whenComplete((__, e) -> log(e, serviceUnit, null, null));
+ }
+
+ private CompletableFuture<MessageId> pubAsync(String serviceUnit, ServiceUnitStateData data) {
+ validateChannelState(Started, true);
+ CompletableFuture<MessageId> future = new CompletableFuture<>();
+ producer.newMessage()
+ .key(serviceUnit)
+ .value(data)
+ .sendAsync()
+ .whenComplete((messageId, e) -> {
+ if (e != null) {
+ log.error("Failed to publish the message: serviceUnit:{}, data:{}",
+ serviceUnit, data, e);
+ future.completeExceptionally(e);
+ } else {
+ future.complete(messageId);
+ }
+ });
+ return future;
+ }
+
+ private CompletableFuture<MessageId> tombstoneAsync(String serviceUnit) {
+ return pubAsync(serviceUnit, null);
+ }
+
+ private boolean isTargetBroker(String broker) {
+ if (broker == null) {
+ return false;
+ }
+ return broker.equals(lookupServiceAddress);
+ }
+
+ private NamespaceBundle getNamespaceBundle(String bundle) {
+ final String namespaceName = LoadManagerShared.getNamespaceNameFromBundleName(bundle);
+ final String bundleRange = LoadManagerShared.getBundleRangeFromBundleName(bundle);
+ return pulsar.getNamespaceService().getNamespaceBundleFactory().getBundle(namespaceName, bundleRange);
+ }
+
+ private CompletableFuture<String> deferGetOwnerRequest(String serviceUnit) {
+ return getOwnerRequests
+ .computeIfAbsent(serviceUnit, k -> {
+ CompletableFuture future = new CompletableFuture<>();
+ future.orTimeout(inFlightStateWaitingTimeInMillis, TimeUnit.MILLISECONDS)
+ .whenComplete((v, e) -> {
+ if (e != null) {
+ getOwnerRequests.remove(serviceUnit);
+ log.warn("Failed to getOwner for serviceUnit:{}",
+ serviceUnit, e);
+ future.completeExceptionally((Throwable) e);
+ }
+ }
+ );
+ return future;
+ });
+ }
+
+ private CompletableFuture<Integer> closeServiceUnit(String serviceUnit) {
+ long startTime = System.nanoTime();
+ MutableInt unloadedTopics = new MutableInt();
+ NamespaceBundle bundle = getNamespaceBundle(serviceUnit);
+ return pulsar.getBrokerService().unloadServiceUnit(
+ bundle,
+ false,
+ pulsar.getConfig().getNamespaceBundleUnloadingTimeoutMs(),
+ TimeUnit.MILLISECONDS)
+ .handle((numUnloadedTopics, ex) -> {
+ if (ex != null) {
+ // ignore topic-close failure to unload bundle
+ log.error("Failed to close topics under bundle:{}",
+ bundle.toString(), ex);
+ } else {
+ unloadedTopics.setValue(numUnloadedTopics);
+ }
+ // clean up topics that failed to unload from the broker ownership cache
+ pulsar.getBrokerService().cleanUnloadedTopicFromCache(bundle);
+ return numUnloadedTopics;
+ })
+ .whenComplete((ignored, ex) -> {
+ double unloadBundleTime = TimeUnit.NANOSECONDS
+ .toMillis((System.nanoTime() - startTime));
+ log.info("Unloading bundle:{} with {} topics completed in {} ms",
+ bundle, unloadedTopics, unloadBundleTime, ex);
+ });
+ }
+
+ private CompletableFuture<Void> splitServiceUnit(String serviceUnit) {
+ // TODO: after the split we need to write the child ownerships to BSC instead of ZK.
+ // Use the configured NamespaceBundleSplitAlgorithm
+ long startTime = System.nanoTime();
+ return pulsar.getNamespaceService()
+ .splitAndOwnBundle(getNamespaceBundle(serviceUnit),
+ false,
+ NamespaceBundleSplitAlgorithm.RANGE_EQUALLY_DIVIDE_ALGO,
+ null)
+ .whenComplete((__, ex) -> {
+ double splitBundleTime = TimeUnit.NANOSECONDS
+ .toMillis((System.nanoTime() - startTime));
+ if (ex == null) {
+ log.info("Successfully split {} namespace-bundle in {} ms",
+ serviceUnit, splitBundleTime);
+ } else {
+ log.error("Failed to split {} namespace-bundle in {} ms",
+ serviceUnit, splitBundleTime, ex);
+ }
+ });
+ }
+
+ public void handleMetadataSessionEvent(SessionEvent e) {
+ if (e == SessionReestablished || e == SessionLost) {
+ lastMetadataSessionEvent = e;
+ lastMetadataSessionEventTimestamp = System.currentTimeMillis();
+ log.info("Received metadata session event:{} at timestamp:{}",
+ lastMetadataSessionEvent, lastMetadataSessionEventTimestamp);
+ }
+ }
+
+ public void handleBrokerRegistrationEvent(String broker, NotificationType type) {
+ if (type == NotificationType.Created) {
+ handleBrokerCreationEvent(broker);
+ } else if (type == NotificationType.Deleted) {
+ handleBrokerDeletionEvent(broker);
+ }
+ }
+
+ private MetadataState getMetadataState() {
+ long now = System.currentTimeMillis();
+ if (lastMetadataSessionEvent == SessionReestablished) {
+ if (now - lastMetadataSessionEventTimestamp > 1000 * maxCleanupDelayTimeInSecs) {
+ return Stable;
+ }
+ return Jittery;
+ }
+ return Unstable;
+ }
+
+ private void handleBrokerCreationEvent(String broker) {
+ CompletableFuture<Void> future = cleanupJobs.remove(broker);
+ if (future != null) {
+ future.cancel(false);
+ totalCleanupCancelledCnt++;
+ log.info("Successfully cancelled the ownership cleanup for broker:{}. isChannelOwner:{}."
+ + " Active cleanup job count:{}",
+ broker, isChannelOwner(), cleanupJobs.size());
+ } else {
+ log.info("Failed to cancel the ownership cleanup for broker:{}. isChannelOwner:{}."
+ + " There was no scheduled cleanup job. Active cleanup job count:{}",
+ broker, isChannelOwner(), cleanupJobs.size());
+ }
+ }
+
+ private void handleBrokerDeletionEvent(String broker) {
+ if (!isChannelOwner()) {
+ return;
+ }
+ MetadataState state = getMetadataState();
+ log.info("Handling broker:{} ownership cleanup based on metadata connection state:{}, event:{}, event_ts:{}:",
+ broker, state, lastMetadataSessionEvent, lastMetadataSessionEventTimestamp);
+ switch (state) {
+ case Stable -> scheduleCleanup(broker, minCleanupDelayTimeInSecs);
+ case Jittery -> scheduleCleanup(broker, maxCleanupDelayTimeInSecs);
+ case Unstable -> {
+ totalCleanupIgnoredCnt++;
+ log.error("MetadataState state is unstable. "
+ + "Ignoring the ownership cleanup request for the reported broker :{}", broker);
+ }
+ }
+ }
+
+ private void scheduleCleanup(String broker, long delayInSecs) {
+ cleanupJobs.computeIfAbsent(broker, k -> {
+ Executor delayed = CompletableFuture
+ .delayedExecutor(delayInSecs, TimeUnit.SECONDS, pulsar.getLoadManagerExecutor());
+ totalCleanupScheduledCnt++;
+ return CompletableFuture
+ .runAsync(() -> doCleanup(broker), delayed);
+ });
+
+ log.info("Scheduled ownership cleanup for broker:{} with delay:{} secs. Pending clean jobs:{}.",
+ broker, delayInSecs, cleanupJobs.size());
+ }
+
+
+ private void doCleanup(String broker) {
+ long startTime = System.nanoTime();
+ log.info("Started ownership cleanup for the inactive broker:{}", broker);
+ AtomicInteger serviceUnitTombstoneCnt = new AtomicInteger();
+ AtomicInteger serviceUnitTombstoneErrorCnt = new AtomicInteger();
+ for (Map.Entry<String, ServiceUnitStateData> etr : tableview.entrySet()) {
+ ServiceUnitStateData stateData = etr.getValue();
+ String serviceUnit = etr.getKey();
+ if (StringUtils.equals(broker, stateData.broker())
+ || StringUtils.equals(broker, stateData.sourceBroker())) {
+ try {
+ outstandingCleanupTombstoneMessages.acquire();
Review Comment:
updated.
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,778 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Free;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Closed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Constructed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.LeaderElectionServiceStarted;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Started;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private final Schema<ServiceUnitStateData> schema;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private final String lookupServiceAddress;
+ // TODO: define BrokerRegistry
+ private final Semaphore outstandingCleanupTombstoneMessages;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+ private final LeaderElectionService leaderElectionService;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private ScheduledFuture<?> cleanupTasks;
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+ private volatile ChannelState channelState;
+
+ enum ChannelState {
+ Closed(0),
+ Constructed(1),
+ LeaderElectionServiceStarted(2),
+ Started(3);
+
+ ChannelState(int id) {
+ this.id = id;
+ }
+ int id;
+ }
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ this.lookupServiceAddress = pulsar.getLookupServiceAddress();
+ this.schema = Schema.JSON(ServiceUnitStateData.class);
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder().build();
+ this.cleanupJobs = ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder().build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} is the leader now.", lookupServiceAddress);
+ // TODO: schedule monitorOwnerships by brokerRegistry
+ } else {
+ log.debug("This broker:{} is a follower now.", lookupServiceAddress);
+ // TODO: cancel scheduled monitorOwnerships if any
+ }
+ });
+ this.channelState = Constructed;
+ }
+
+ public synchronized void start() throws PulsarServerException {
+ validateChannelState(LeaderElectionServiceStarted, false);
+ try {
+ leaderElectionService.start();
+ this.channelState = LeaderElectionServiceStarted;
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(schema)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(schema)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ channelState = Started;
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public synchronized void close() throws PulsarServerException {
+ channelState = Closed;
+ try {
+ leaderElectionService.close();
+ log.debug("Successfully closed the channel leader election service.");
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ private void validateChannelState(ChannelState targetState, boolean checkLowerIds) {
+ int order = checkLowerIds ? -1 : 1;
+ if (Integer.compare(channelState.id, targetState.id) * order > 0) {
+ throw new IllegalStateException("Invalid channel state:" + channelState.name());
+ }
+ }
+
+ public synchronized void scheduleCompaction() throws PulsarServerException {
+ try {
+ Long threshold = pulsar.getAdminClient().topicPolicies()
+ .getCompactionThreshold(TOPIC);
+ if (threshold == null || threshold == 0) {
+ pulsar.getAdminClient().topicPolicies()
+ .setCompactionThreshold(TOPIC, COMPACTION_THRESHOLD);
+ log.info("Scheduled compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ } else {
+ log.info("Already set compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ }
+ } catch (PulsarAdminException e) {
+ throw new PulsarServerException("Failed to schedule compaction.", e);
+ }
+ }
+
+ public String getChannelOwner() {
+ validateChannelState(LeaderElectionServiceStarted, true);
+ CompletableFuture<Optional<LeaderBroker>> future = leaderElectionService.readCurrentLeader();
+ if (!future.isDone()) {
+ return null;
+ }
+ Optional<LeaderBroker> leader = future.join();
+ if (leader.isEmpty()) {
+ return null;
+ }
+ //expecting http://broker-xyz:port
+ // TODO: discard this protocol prefix removal by a util func that returns lookupServiceAddress(serviceUrl)
+ String broker = leader.get().getServiceUrl();
+ broker = broker.substring(broker.lastIndexOf('/') + 1);
+ return broker;
+ }
+
+ public boolean isChannelOwner() {
+ return isTargetBroker(getChannelOwner());
+ }
+
+ public CompletableFuture<String> getOwnerAsync(String serviceUnit) {
+ validateChannelState(Started, true);
+ ServiceUnitStateData data = tableview.get(serviceUnit);
+ if (data == null) {
+ return null;
+ }
+ switch (data.state()) {
+ case Owned, Splitting -> {
+ return CompletableFuture.completedFuture(data.broker());
+ }
+ case Assigned, Released -> {
+ return deferGetOwnerRequest(serviceUnit);
+ }
+ default -> {
+ return null;
+ }
+ }
+ }
+
+ public CompletableFuture<String> publishAssignEventAsync(String serviceUnit, String broker) {
+ CompletableFuture<String> getOwnerRequest = deferGetOwnerRequest(serviceUnit);
+ pubAsync(serviceUnit, new ServiceUnitStateData(Assigned, broker))
+ .whenComplete((__, ex) -> {
+ if (ex != null) {
+ getOwnerRequests.remove(serviceUnit);
+ if (!getOwnerRequest.isCompletedExceptionally()) {
+ getOwnerRequest.completeExceptionally(ex);
+ }
+ }
+ });
+
+ return getOwnerRequest;
+ }
+
+ public CompletableFuture<Void> publishUnloadEventAsync(Unload unload) {
+ String serviceUnit = unload.serviceUnit();
+ if (isTransferCommand(unload)) {
+ ServiceUnitStateData next = new ServiceUnitStateData(Assigned,
+ unload.destBroker().get(), unload.sourceBroker());
+ return pubAsync(serviceUnit, next).thenAccept(__ -> {
+ });
Review Comment:
updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] heesung-sn commented on a diff in pull request #18489: [improve][broker] PIP-192 Added ServiceUnitStateChannelImpl
Posted by GitBox <gi...@apache.org>.
heesung-sn commented on code in PR #18489:
URL: https://github.com/apache/pulsar/pull/18489#discussion_r1034310909
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,778 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Free;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Closed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Constructed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.LeaderElectionServiceStarted;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Started;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private final Schema<ServiceUnitStateData> schema;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private final String lookupServiceAddress;
+ // TODO: define BrokerRegistry
+ private final Semaphore outstandingCleanupTombstoneMessages;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+ private final LeaderElectionService leaderElectionService;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private ScheduledFuture<?> cleanupTasks;
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+ private volatile ChannelState channelState;
+
+ enum ChannelState {
+ Closed(0),
+ Constructed(1),
+ LeaderElectionServiceStarted(2),
+ Started(3);
+
+ ChannelState(int id) {
+ this.id = id;
+ }
+ int id;
+ }
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ this.lookupServiceAddress = pulsar.getLookupServiceAddress();
+ this.schema = Schema.JSON(ServiceUnitStateData.class);
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder().build();
+ this.cleanupJobs = ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder().build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} is the leader now.", lookupServiceAddress);
+ // TODO: schedule monitorOwnerships by brokerRegistry
+ } else {
+ log.debug("This broker:{} is a follower now.", lookupServiceAddress);
+ // TODO: cancel scheduled monitorOwnerships if any
+ }
+ });
+ this.channelState = Constructed;
+ }
+
+ public synchronized void start() throws PulsarServerException {
+ validateChannelState(LeaderElectionServiceStarted, false);
+ try {
+ leaderElectionService.start();
+ this.channelState = LeaderElectionServiceStarted;
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(schema)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(schema)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ channelState = Started;
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public synchronized void close() throws PulsarServerException {
+ channelState = Closed;
+ try {
+ leaderElectionService.close();
+ log.debug("Successfully closed the channel leader election service.");
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ private void validateChannelState(ChannelState targetState, boolean checkLowerIds) {
+ int order = checkLowerIds ? -1 : 1;
+ if (Integer.compare(channelState.id, targetState.id) * order > 0) {
+ throw new IllegalStateException("Invalid channel state:" + channelState.name());
+ }
+ }
+
+ public synchronized void scheduleCompaction() throws PulsarServerException {
Review Comment:
From my understanding, this needs to be called separately when the topic policy is ready.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] shibd commented on a diff in pull request #18489: [improve][broker] PIP-192 Added ServiceUnitStateChannelImpl
Posted by GitBox <gi...@apache.org>.
shibd commented on code in PR #18489:
URL: https://github.com/apache/pulsar/pull/18489#discussion_r1027282225
##########
pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelTest.java:
##########
@@ -0,0 +1,772 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.ConnectionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.Reconnected;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.greaterThanOrEqualTo;
+import static org.hamcrest.Matchers.lessThanOrEqualTo;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.assertNull;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.eq;
+import static org.mockito.Mockito.doReturn;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.spy;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.testng.Assert.assertFalse;
+import static org.testng.Assert.assertTrue;
+import static org.testng.AssertJUnit.assertNotNull;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import org.apache.commons.lang.StringUtils;
+import org.apache.commons.lang3.reflect.FieldUtils;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.auth.MockedPulsarServiceBaseTest;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.TypedMessageBuilder;
+import org.apache.pulsar.client.impl.TableViewImpl;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+import org.awaitility.Awaitility;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.Test;
+
+@Test(groups = "broker")
+public class ServiceUnitStateChannelTest extends MockedPulsarServiceBaseTest {
+
+ private PulsarService pulsar1;
+ private PulsarService pulsar2;
+ private ServiceUnitStateChannel channel1;
+ private ServiceUnitStateChannel channel2;
+ private String lookupServiceAddress1;
+ private String lookupServiceAddress2;
+ private String bundle;
+
+ private String bundle1;
+ private String bundle2;
+
+ @BeforeClass
+ @Override
+ protected void setup() throws Exception {
+ conf.setAllowAutoTopicCreation(true);
+ super.internalSetup(conf);
+
+ admin.tenants().createTenant("pulsar", createDefaultTenantInfo());
+ admin.namespaces().createNamespace("pulsar/system");
+ admin.tenants().createTenant("public", createDefaultTenantInfo());
+ admin.namespaces().createNamespace("public/default");
+
+ pulsar1 = pulsar;
+ pulsar2 = startBrokerWithoutAuthorization(getDefaultConf());
+ channel1 = new ServiceUnitStateChannelImpl(pulsar1);
+ channel1.start();
+ channel2 = new ServiceUnitStateChannelImpl(pulsar2);
+ channel2.start();
+ lookupServiceAddress1 = (String)
+ FieldUtils.readDeclaredField(channel1, "lookupServiceAddress", true);
+ lookupServiceAddress2 = (String)
+ FieldUtils.readDeclaredField(channel2, "lookupServiceAddress", true);
+
+ bundle = String.format("%s/%s", "public/default", "0x00000000_0xffffffff");
+ bundle1 = String.format("%s/%s", "public/default", "0x00000000_0xfffffff0");
+ bundle2 = String.format("%s/%s", "public/default", "0xfffffff0_0xffffffff");
+ }
+
+ @BeforeMethod
+ protected void initTableViews() throws Exception {
+ cleanTableView(channel1, bundle);
+ cleanTableView(channel2, bundle);
+ }
+
+
+ @AfterClass
+ @Override
+ protected void cleanup() throws Exception {
+ channel1.close();
+ channel2.close();
+ pulsar1 = null;
+ pulsar2.close();
+ super.internalCleanup();
+ }
+
+ @Test(priority = 0)
+ public void channelOwnerTest() throws Exception {
+ var channelOwner1 = channel1.getChannelOwner();
+ var channelOwner2 = channel2.getChannelOwner();
+ assertEquals(channelOwner1, channelOwner2);
+ LeaderElectionService leaderElectionService1 = (LeaderElectionService) FieldUtils.readDeclaredField(
+ channel1, "leaderElectionService", true);
+ leaderElectionService1.close();
+ waitUntilNewChannelOwner(channel2, channelOwner1);
+ leaderElectionService1.start();
+ waitUntilNewChannelOwner(channel1, channelOwner1);
+
+ var newChannelOwner1 = channel1.getChannelOwner();
+ var newChannelOwner2 = channel2.getChannelOwner();
+
+ assertEquals(newChannelOwner1, newChannelOwner2);
+ assertNotEquals(channelOwner1, newChannelOwner1);
+
+ if (newChannelOwner1.equals(lookupServiceAddress1)) {
+ assertTrue(channel1.isChannelOwner());
+ assertFalse(channel2.isChannelOwner());
+ } else {
+ assertFalse(channel1.isChannelOwner());
+ assertTrue(channel2.isChannelOwner());
+ }
+ }
+
+ @Test(priority = 0)
+ public void channelValidationTest() throws ExecutionException, InterruptedException {
+ var channel = new ServiceUnitStateChannelImpl(pulsar);
+
+ int errorCnt = validateChannelStart(channel);
+ assertEquals(6, errorCnt);
+ ExecutorService executor = Executors.newSingleThreadExecutor();
+ Future startFuture = executor.submit(() -> {
+ try {
+ channel.start();
+ } catch (PulsarServerException e) {
+ throw new RuntimeException(e);
+ }
+ });
+ errorCnt = validateChannelStart(channel);
+ startFuture.get();
+ assertTrue(errorCnt > 0);
+ Future closeFuture = executor.submit(()->{
+ try {
+ channel.close();
+ } catch (PulsarServerException e) {
+ throw new RuntimeException(e);
+ }
+ });
+ errorCnt = validateChannelStart(channel);
+ closeFuture.get();
+ assertTrue(errorCnt > 0);
+ errorCnt = validateChannelStart(channel);
+ assertEquals(6, errorCnt);
+ }
+
+ private int validateChannelStart(ServiceUnitStateChannelImpl channel) {
+ int errorCnt = 0;
+ try {
+ channel.isChannelOwner();
+ } catch (IllegalStateException e) {
+ errorCnt++;
+ }
+ try {
+ channel.getChannelOwner();
+ } catch (IllegalStateException e) {
+ errorCnt++;
+ }
+ try {
+ channel.getOwnerAsync(bundle);
+ } catch (IllegalStateException e) {
+ errorCnt++;
+ }
+ try {
+ channel.publishAssignEventAsync(bundle, lookupServiceAddress1);
+ } catch (IllegalStateException e) {
+ errorCnt++;
+ }
+ try {
+ channel.publishUnloadEventAsync(
+ new Unload(lookupServiceAddress1, bundle, Optional.of(lookupServiceAddress2)));
+ } catch (IllegalStateException e) {
+ errorCnt++;
+ }
+ try {
+ channel.publishSplitEventAsync(new Split(bundle, lookupServiceAddress1, Map.of()));
+ } catch (IllegalStateException e) {
+ errorCnt++;
+ }
+ return errorCnt;
+ }
+
+ @Test(priority = 1)
+ public void compactionScheduleTest() throws PulsarServerException {
+
+ channel1.scheduleCompaction();
+
+ Awaitility.await()
+ .pollInterval(200, TimeUnit.MILLISECONDS)
+ .atMost(5, TimeUnit.SECONDS)
+ .untilAsserted(() -> { // wait until true
+ try {
+ var threshold = admin.topicPolicies()
+ .getCompactionThreshold(ServiceUnitStateChannelImpl.TOPIC, false).longValue();
+ assertEquals(5 * 1024 * 1024, threshold);
+ } catch (Exception e) {
+ ;
+ }
+
+ });
+ }
+
+ @Test(priority = 2)
+ public void assignmentTest()
+ throws ExecutionException, InterruptedException, IllegalAccessException, TimeoutException {
+ var getOwnerRequests1 = spy(getOwnerRequests((channel1)));
+ var getOwnerRequests2 = spy(getOwnerRequests((channel2)));
+
+ var owner1 = channel1.getOwnerAsync(bundle);
+ var owner2 = channel2.getOwnerAsync(bundle);
+
+ assertNull(owner1);
+ assertNull(owner2);
+
+ var assigned1 = channel1.publishAssignEventAsync(bundle, lookupServiceAddress1);
+ var assigned2 = channel2.publishAssignEventAsync(bundle, lookupServiceAddress2);
+ assertNotNull(assigned1);
+ assertNotNull(assigned2);
+ waitUntilOwnerChanges(channel2, bundle, null);
Review Comment:
> I think adding the `Command` layer is additional complexity and requires more lines of code. Also, the current approach does not need to store the additional `Command` info in the system topic(space optimized).
>
> Regarding `ServiceUnitState.isValidTransition()`, the state validation(aka compaction) will happen in the tableview and topic compaction by `TopicCompactionStrategy`(which is in a separate PIP and PR here, #18099). This validation logic, ServiceUnitChannelCompactionStrategy will be added after this PR #18195 gets merged.
Thanks for the explanation, I'm not quite sure if my proposal is reasonable. I'll wait for other features to be added and revisit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] shibd commented on a diff in pull request #18489: [improve][broker] PIP-192 Added ServiceUnitStateChannelImpl
Posted by GitBox <gi...@apache.org>.
shibd commented on code in PR #18489:
URL: https://github.com/apache/pulsar/pull/18489#discussion_r1029961985
##########
pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelTest.java:
##########
@@ -0,0 +1,796 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.ConnectionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.Reconnected;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.greaterThanOrEqualTo;
+import static org.hamcrest.Matchers.lessThanOrEqualTo;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.assertNull;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.eq;
+import static org.mockito.Mockito.doReturn;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.spy;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.testng.Assert.assertFalse;
+import static org.testng.Assert.assertTrue;
+import static org.testng.AssertJUnit.assertNotNull;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import org.apache.commons.lang.StringUtils;
+import org.apache.commons.lang3.reflect.FieldUtils;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.auth.MockedPulsarServiceBaseTest;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.TypedMessageBuilder;
+import org.apache.pulsar.client.impl.TableViewImpl;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+import org.awaitility.Awaitility;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.Test;
+
+@Test(groups = "broker")
+public class ServiceUnitStateChannelTest extends MockedPulsarServiceBaseTest {
+
+ private PulsarService pulsar1;
+ private PulsarService pulsar2;
+ private ServiceUnitStateChannel channel1;
+ private ServiceUnitStateChannel channel2;
+ private String lookupServiceAddress1;
+ private String lookupServiceAddress2;
+ private String bundle;
+
+ private String bundle1;
+ private String bundle2;
+
+ @BeforeClass
+ @Override
+ protected void setup() throws Exception {
+ conf.setAllowAutoTopicCreation(true);
+ super.internalSetup(conf);
+
+ admin.tenants().createTenant("pulsar", createDefaultTenantInfo());
+ admin.namespaces().createNamespace("pulsar/system");
+ admin.tenants().createTenant("public", createDefaultTenantInfo());
+ admin.namespaces().createNamespace("public/default");
+
+ pulsar1 = pulsar;
+ pulsar2 = startBrokerWithoutAuthorization(getDefaultConf());
+ channel1 = new ServiceUnitStateChannelImpl(pulsar1);
+ channel1.start();
+ channel2 = new ServiceUnitStateChannelImpl(pulsar2);
+ channel2.start();
+ lookupServiceAddress1 = (String)
+ FieldUtils.readDeclaredField(channel1, "lookupServiceAddress", true);
+ lookupServiceAddress2 = (String)
+ FieldUtils.readDeclaredField(channel2, "lookupServiceAddress", true);
+
+ bundle = String.format("%s/%s", "public/default", "0x00000000_0xffffffff");
+ bundle1 = String.format("%s/%s", "public/default", "0x00000000_0xfffffff0");
+ bundle2 = String.format("%s/%s", "public/default", "0xfffffff0_0xffffffff");
+ }
+
+ @BeforeMethod
+ protected void initTableViews() throws Exception {
+ cleanTableView(channel1, bundle);
+ cleanTableView(channel2, bundle);
+ }
+
+
+ @AfterClass
+ @Override
+ protected void cleanup() throws Exception {
+ channel1.close();
+ channel2.close();
+ pulsar1 = null;
+ pulsar2.close();
+ super.internalCleanup();
+ }
+
+ @Test(priority = 0)
+ public void channelOwnerTest() throws Exception {
+ var channelOwner1 = channel1.getChannelOwner();
+ var channelOwner2 = channel2.getChannelOwner();
+ assertEquals(channelOwner1, channelOwner2);
+ LeaderElectionService leaderElectionService1 = (LeaderElectionService) FieldUtils.readDeclaredField(
+ channel1, "leaderElectionService", true);
+ leaderElectionService1.close();
+ waitUntilNewChannelOwner(channel2, channelOwner1);
+ leaderElectionService1.start();
+ waitUntilNewChannelOwner(channel1, channelOwner1);
+
+ var newChannelOwner1 = channel1.getChannelOwner();
+ var newChannelOwner2 = channel2.getChannelOwner();
+
+ assertEquals(newChannelOwner1, newChannelOwner2);
+ assertNotEquals(channelOwner1, newChannelOwner1);
+
+ if (newChannelOwner1.equals(lookupServiceAddress1)) {
+ assertTrue(channel1.isChannelOwner());
+ assertFalse(channel2.isChannelOwner());
+ } else {
+ assertFalse(channel1.isChannelOwner());
+ assertTrue(channel2.isChannelOwner());
+ }
+ }
+
+ @Test(priority = 0)
+ public void channelValidationTest()
+ throws ExecutionException, InterruptedException, IllegalAccessException, PulsarServerException {
+ var channel = new ServiceUnitStateChannelImpl(pulsar);
+ int errorCnt = validateChannelStart(channel);
+ assertEquals(6, errorCnt);
+ ExecutorService executor = Executors.newSingleThreadExecutor();
+ Future startFuture = executor.submit(() -> {
+ try {
+ channel.start();
+ } catch (PulsarServerException e) {
+ throw new RuntimeException(e);
+ }
+ });
+ errorCnt = validateChannelStart(channel);
+ startFuture.get();
+ assertTrue(errorCnt > 0);
+
+ FieldUtils.writeDeclaredField(channel, "channelState",
+ ServiceUnitStateChannelImpl.ChannelState.LeaderElectionServiceStarted, true);
+ assertNotNull(channel.getChannelOwner());
+
+ Future closeFuture = executor.submit(()->{
+ try {
+ channel.close();
+ } catch (PulsarServerException e) {
+ throw new RuntimeException(e);
+ }
+ });
+ errorCnt = validateChannelStart(channel);
+ closeFuture.get();
+ assertTrue(errorCnt > 0);
+ errorCnt = validateChannelStart(channel);
+ assertEquals(6, errorCnt);
+
+ // check if we can close() again
+ channel.close();
+ errorCnt = validateChannelStart(channel);
+ assertEquals(6, errorCnt);
+
+ // close() -> start() test.
+ channel.start();
+ assertNotNull(channel.getChannelOwner());
+
+ // start() -> start() test.
+ IllegalStateException ex = null;
Review Comment:
Suggestion use: `assertThrows(IllegalStateException.class, () -> channel.start());`
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,778 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Free;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Closed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Constructed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.LeaderElectionServiceStarted;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Started;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private final Schema<ServiceUnitStateData> schema;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private final String lookupServiceAddress;
+ // TODO: define BrokerRegistry
+ private final Semaphore outstandingCleanupTombstoneMessages;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+ private final LeaderElectionService leaderElectionService;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private ScheduledFuture<?> cleanupTasks;
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+ private volatile ChannelState channelState;
+
+ enum ChannelState {
+ Closed(0),
+ Constructed(1),
+ LeaderElectionServiceStarted(2),
+ Started(3);
+
+ ChannelState(int id) {
+ this.id = id;
+ }
+ int id;
+ }
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ this.lookupServiceAddress = pulsar.getLookupServiceAddress();
+ this.schema = Schema.JSON(ServiceUnitStateData.class);
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder().build();
+ this.cleanupJobs = ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder().build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} is the leader now.", lookupServiceAddress);
+ // TODO: schedule monitorOwnerships by brokerRegistry
+ } else {
+ log.debug("This broker:{} is a follower now.", lookupServiceAddress);
+ // TODO: cancel scheduled monitorOwnerships if any
+ }
+ });
+ this.channelState = Constructed;
+ }
+
+ public synchronized void start() throws PulsarServerException {
+ validateChannelState(LeaderElectionServiceStarted, false);
+ try {
+ leaderElectionService.start();
+ this.channelState = LeaderElectionServiceStarted;
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(schema)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(schema)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ channelState = Started;
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public synchronized void close() throws PulsarServerException {
+ channelState = Closed;
+ try {
+ leaderElectionService.close();
+ log.debug("Successfully closed the channel leader election service.");
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ private void validateChannelState(ChannelState targetState, boolean checkLowerIds) {
+ int order = checkLowerIds ? -1 : 1;
+ if (Integer.compare(channelState.id, targetState.id) * order > 0) {
+ throw new IllegalStateException("Invalid channel state:" + channelState.name());
+ }
+ }
+
+ public synchronized void scheduleCompaction() throws PulsarServerException {
+ try {
+ Long threshold = pulsar.getAdminClient().topicPolicies()
+ .getCompactionThreshold(TOPIC);
+ if (threshold == null || threshold == 0) {
+ pulsar.getAdminClient().topicPolicies()
+ .setCompactionThreshold(TOPIC, COMPACTION_THRESHOLD);
+ log.info("Scheduled compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ } else {
+ log.info("Already set compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ }
+ } catch (PulsarAdminException e) {
+ throw new PulsarServerException("Failed to schedule compaction.", e);
+ }
+ }
+
+ public String getChannelOwner() {
+ validateChannelState(LeaderElectionServiceStarted, true);
+ CompletableFuture<Optional<LeaderBroker>> future = leaderElectionService.readCurrentLeader();
+ if (!future.isDone()) {
Review Comment:
When the future is not done, we should wait for it, right? We should remove 255~257, otherwise unit test `channelValidationTest` doesn't pass.
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,778 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Free;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Closed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Constructed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.LeaderElectionServiceStarted;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Started;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private final Schema<ServiceUnitStateData> schema;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private final String lookupServiceAddress;
+ // TODO: define BrokerRegistry
+ private final Semaphore outstandingCleanupTombstoneMessages;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+ private final LeaderElectionService leaderElectionService;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private ScheduledFuture<?> cleanupTasks;
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+ private volatile ChannelState channelState;
+
+ enum ChannelState {
+ Closed(0),
+ Constructed(1),
+ LeaderElectionServiceStarted(2),
+ Started(3);
+
+ ChannelState(int id) {
+ this.id = id;
+ }
+ int id;
+ }
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ this.lookupServiceAddress = pulsar.getLookupServiceAddress();
+ this.schema = Schema.JSON(ServiceUnitStateData.class);
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder().build();
+ this.cleanupJobs = ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder().build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} is the leader now.", lookupServiceAddress);
+ // TODO: schedule monitorOwnerships by brokerRegistry
+ } else {
+ log.debug("This broker:{} is a follower now.", lookupServiceAddress);
+ // TODO: cancel scheduled monitorOwnerships if any
+ }
+ });
+ this.channelState = Constructed;
+ }
+
+ public synchronized void start() throws PulsarServerException {
+ validateChannelState(LeaderElectionServiceStarted, false);
+ try {
+ leaderElectionService.start();
+ this.channelState = LeaderElectionServiceStarted;
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(schema)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(schema)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ channelState = Started;
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public synchronized void close() throws PulsarServerException {
+ channelState = Closed;
+ try {
+ leaderElectionService.close();
+ log.debug("Successfully closed the channel leader election service.");
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ private void validateChannelState(ChannelState targetState, boolean checkLowerIds) {
+ int order = checkLowerIds ? -1 : 1;
+ if (Integer.compare(channelState.id, targetState.id) * order > 0) {
+ throw new IllegalStateException("Invalid channel state:" + channelState.name());
+ }
+ }
+
+ public synchronized void scheduleCompaction() throws PulsarServerException {
Review Comment:
Who called this method? It looks like we only need to call it once at the start.
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,778 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Free;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Closed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Constructed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.LeaderElectionServiceStarted;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Started;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private final Schema<ServiceUnitStateData> schema;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private final String lookupServiceAddress;
+ // TODO: define BrokerRegistry
+ private final Semaphore outstandingCleanupTombstoneMessages;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+ private final LeaderElectionService leaderElectionService;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private ScheduledFuture<?> cleanupTasks;
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+ private volatile ChannelState channelState;
+
+ enum ChannelState {
+ Closed(0),
+ Constructed(1),
+ LeaderElectionServiceStarted(2),
+ Started(3);
+
+ ChannelState(int id) {
+ this.id = id;
+ }
+ int id;
+ }
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ this.lookupServiceAddress = pulsar.getLookupServiceAddress();
+ this.schema = Schema.JSON(ServiceUnitStateData.class);
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder().build();
+ this.cleanupJobs = ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder().build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} is the leader now.", lookupServiceAddress);
+ // TODO: schedule monitorOwnerships by brokerRegistry
+ } else {
+ log.debug("This broker:{} is a follower now.", lookupServiceAddress);
+ // TODO: cancel scheduled monitorOwnerships if any
+ }
+ });
+ this.channelState = Constructed;
+ }
+
+ public synchronized void start() throws PulsarServerException {
+ validateChannelState(LeaderElectionServiceStarted, false);
+ try {
+ leaderElectionService.start();
+ this.channelState = LeaderElectionServiceStarted;
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(schema)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(schema)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ channelState = Started;
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public synchronized void close() throws PulsarServerException {
+ channelState = Closed;
+ try {
+ leaderElectionService.close();
+ log.debug("Successfully closed the channel leader election service.");
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ private void validateChannelState(ChannelState targetState, boolean checkLowerIds) {
+ int order = checkLowerIds ? -1 : 1;
+ if (Integer.compare(channelState.id, targetState.id) * order > 0) {
+ throw new IllegalStateException("Invalid channel state:" + channelState.name());
+ }
+ }
+
+ public synchronized void scheduleCompaction() throws PulsarServerException {
+ try {
+ Long threshold = pulsar.getAdminClient().topicPolicies()
+ .getCompactionThreshold(TOPIC);
+ if (threshold == null || threshold == 0) {
+ pulsar.getAdminClient().topicPolicies()
+ .setCompactionThreshold(TOPIC, COMPACTION_THRESHOLD);
+ log.info("Scheduled compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ } else {
+ log.info("Already set compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
Review Comment:
```suggestion
log.info("Already set compaction on topic:{}, threshold:{} bytes", TOPIC, threshold);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] heesung-sn commented on a diff in pull request #18489: [improve][broker] PIP-192 Added ServiceUnitStateChannelImpl
Posted by GitBox <gi...@apache.org>.
heesung-sn commented on code in PR #18489:
URL: https://github.com/apache/pulsar/pull/18489#discussion_r1034352125
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,778 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Free;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Closed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Constructed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.LeaderElectionServiceStarted;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Started;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private final Schema<ServiceUnitStateData> schema;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private final String lookupServiceAddress;
+ // TODO: define BrokerRegistry
+ private final Semaphore outstandingCleanupTombstoneMessages;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+ private final LeaderElectionService leaderElectionService;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private ScheduledFuture<?> cleanupTasks;
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+ private volatile ChannelState channelState;
+
+ enum ChannelState {
+ Closed(0),
+ Constructed(1),
+ LeaderElectionServiceStarted(2),
+ Started(3);
+
+ ChannelState(int id) {
+ this.id = id;
+ }
+ int id;
+ }
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ this.lookupServiceAddress = pulsar.getLookupServiceAddress();
+ this.schema = Schema.JSON(ServiceUnitStateData.class);
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder().build();
+ this.cleanupJobs = ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder().build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} is the leader now.", lookupServiceAddress);
+ // TODO: schedule monitorOwnerships by brokerRegistry
+ } else {
+ log.debug("This broker:{} is a follower now.", lookupServiceAddress);
+ // TODO: cancel scheduled monitorOwnerships if any
+ }
+ });
+ this.channelState = Constructed;
+ }
+
+ public synchronized void start() throws PulsarServerException {
+ validateChannelState(LeaderElectionServiceStarted, false);
+ try {
+ leaderElectionService.start();
+ this.channelState = LeaderElectionServiceStarted;
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(schema)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(schema)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ channelState = Started;
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public synchronized void close() throws PulsarServerException {
+ channelState = Closed;
+ try {
+ leaderElectionService.close();
+ log.debug("Successfully closed the channel leader election service.");
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ private void validateChannelState(ChannelState targetState, boolean checkLowerIds) {
+ int order = checkLowerIds ? -1 : 1;
+ if (Integer.compare(channelState.id, targetState.id) * order > 0) {
+ throw new IllegalStateException("Invalid channel state:" + channelState.name());
+ }
+ }
+
+ public synchronized void scheduleCompaction() throws PulsarServerException {
Review Comment:
Ack. Removing this api
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on a diff in pull request #18489: [improve][broker] PIP-192 Added ServiceUnitStateChannelImpl
Posted by GitBox <gi...@apache.org>.
eolivelli commented on code in PR #18489:
URL: https://github.com/apache/pulsar/pull/18489#discussion_r1026183482
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,779 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ private static final Schema<ServiceUnitStateData> SCHEMA = Schema.JSON(ServiceUnitStateData.class);
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private String lookupServiceAddress;
+ private LeaderElectionService leaderElectionService;
+ // TODO: define BrokerRegistry
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private Semaphore outstandingPubMessages;
+
+ private Semaphore outstandingCleanupTombstoneMessages;
+ private ScheduledFuture<?> cleanupTasks;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+
+ private ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ ServiceConfiguration conf = pulsar.getConfiguration();
+ this.lookupServiceAddress = pulsar.getAdvertisedAddress() + ":"
+ + (conf.getWebServicePort().isPresent() ? conf.getWebServicePort().get()
+ : conf.getWebServicePortTls().get());
+ this.outstandingPubMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder()
+ .build();
+ this.cleanupJobs =
+ ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder()
+ .build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ }
+
+ public void start() throws PulsarServerException {
+
+ try {
+ if (leaderElectionService != null) {
+ leaderElectionService.close();
+ log.debug("Closed the channel leader election service.");
+ }
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} was elected as the leader."
+ + "Current channel leader is {}",
+ lookupServiceAddress,
+ leaderElectionService.getCurrentLeader());
+ } else {
+ if (leaderElectionService != null) {
+ log.debug("This broker:{} is a follower. "
+ + "Current channel leader is {}",
+ lookupServiceAddress,
+ leaderElectionService.getCurrentLeader());
+ }
+ }
+ });
+ leaderElectionService.start();
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(SCHEMA)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(SCHEMA)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ // TODO: schedule cleanupTasks by brokerRegistry
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+
+ public void close() throws PulsarServerException {
+ try {
+ if (leaderElectionService != null) {
+ leaderElectionService.close();
+ leaderElectionService = null;
+ log.debug("Successfully closed the channel leader election service.");
+ }
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public void scheduleCompaction() throws PulsarServerException {
+ try {
+ Long threshold = pulsar.getAdminClient().topicPolicies()
+ .getCompactionThreshold(TOPIC);
+ if (threshold == null || threshold == 0) {
+ pulsar.getAdminClient().topicPolicies()
+ .setCompactionThreshold(TOPIC, COMPACTION_THRESHOLD);
+ log.info("Scheduled compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ } else {
+ log.info("Already set compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ }
+ } catch (PulsarAdminException e) {
+ throw new PulsarServerException("Failed to schedule compaction.", e);
+ }
+ }
+
+ public String getChannelOwner() {
+ CompletableFuture<Optional<LeaderBroker>> future = leaderElectionService.readCurrentLeader();
+ if (!future.isDone()) {
+ return null;
+ }
+ Optional<LeaderBroker> leader = future.join();
+ if (leader.isEmpty()) {
+ return null;
+ }
+ //expecting http://broker-xyz:port
+ String broker = leader.get().getServiceUrl();
+ broker = broker.substring(broker.lastIndexOf('/') + 1);
+ return broker;
+ }
+
+ public boolean isChannelOwner() {
+ return isTargetBroker(getChannelOwner());
+ }
+
+ public CompletableFuture<String> getOwnerAsync(String serviceUnit) {
+
+ ServiceUnitStateData data = tableview.get(serviceUnit);
+ if (data == null) {
+ return null;
+ }
+ switch (data.state()) {
+ case Owned, Splitting -> {
+ return CompletableFuture.completedFuture(data.broker());
+ }
+ case Assigned, Released -> {
+ return deferGetOwnerRequest(serviceUnit);
+ }
+ default -> {
+ return null;
+ }
+ }
+ }
+
+ public CompletableFuture<String> publishAssignEventAsync(String serviceUnit, String broker) {
+ CompletableFuture<String> getOwnerRequest = deferGetOwnerRequest(serviceUnit);
+ pubAsync(serviceUnit, new ServiceUnitStateData(Assigned, broker))
+ .whenComplete((__, ex) -> {
+ if (ex != null) {
+ getOwnerRequests.remove(serviceUnit);
+ if (!getOwnerRequest.isCompletedExceptionally()) {
+ getOwnerRequest.completeExceptionally(ex);
+ }
+ }
+ });
+
+ return getOwnerRequest;
+ }
+
+ public CompletableFuture<Void> publishUnloadEventAsync(Unload unload) {
+ String serviceUnit = unload.serviceUnit();
+ if (isTransferCommand(unload)) {
+ ServiceUnitStateData next = new ServiceUnitStateData(Assigned,
+ unload.destBroker().get(), unload.sourceBroker());
+ return pubAsync(serviceUnit, next).thenAccept(__ -> {
+ });
+ }
+ return tombstoneAsync(serviceUnit).thenAccept(__ -> {
+ });
+ }
+
+ public CompletableFuture<Void> publishSplitEventAsync(Split split) {
+ String serviceUnit = split.serviceUnit();
+ ServiceUnitStateData data = tableview.get(serviceUnit);
+ ServiceUnitStateData next = new ServiceUnitStateData(Splitting, data.broker());
+ return pubAsync(serviceUnit, next).thenAccept(__ -> {
+ });
+ }
+
+ private void handle(String serviceUnit, ServiceUnitStateData data) {
+ if (log.isDebugEnabled()) {
+ log.info("{} received a handle request for serviceUnit:{}, data:{}",
+ lookupServiceAddress, serviceUnit, data);
+ }
+
+ if (data == null) {
+ handleTombstoneEvent(serviceUnit);
+ return;
+ }
+
+ // TODO : Add state validation
+ switch (data.state()) {
+ case Owned -> handleOwnEvent(serviceUnit, data);
+ case Assigned -> handleAssignEvent(serviceUnit, data);
+ case Released -> handleReleaseEvent(serviceUnit, data);
+ case Splitting -> handleSplitEvent(serviceUnit, data);
+ default -> throw new IllegalStateException("Failed to handle channel data:" + data);
+ }
+ }
+
+ private static boolean isTransferCommand(ServiceUnitStateData data) {
+ if (data == null) {
+ return false;
+ }
+ return StringUtils.isNotEmpty(data.sourceBroker());
+ }
+
+ private static boolean isTransferCommand(Unload data) {
+ return data.destBroker().isPresent();
+ }
+
+ private static String getLogEventTag(ServiceUnitStateData data) {
+ return data == null ? "Free" :
+ isTransferCommand(data) ? "Transfer:" + data.state() : data.state().toString();
+ }
+
+ private void log(Throwable e, String serviceUnit, ServiceUnitStateData data, ServiceUnitStateData next) {
+ if (e == null) {
+ if (log.isDebugEnabled() || isTransferCommand(data)) {
+ log.info("{} handled {} event for serviceUnit:{}, cur:{}, next:{}",
+ lookupServiceAddress, getLogEventTag(data), serviceUnit,
+ data == null ? "" : data,
+ next == null ? "" : next);
+ }
+ } else {
+ log.error("{} failed to handle {} event for serviceUnit:{}, cur:{}, next:{}",
+ lookupServiceAddress, getLogEventTag(data), serviceUnit,
+ data == null ? "" : data,
+ next == null ? "" : next,
+ e);
+ }
+ }
+
+ private void handleOwnEvent(String serviceUnit, ServiceUnitStateData data) {
+ var getOwnerRequest = getOwnerRequests.remove(serviceUnit);
+ if (getOwnerRequest != null) {
+ getOwnerRequest.complete(data.broker());
+ }
+ if (isTargetBroker(data.broker())) {
+ log(null, serviceUnit, data, null);
+ }
+
+ }
+
+ private void handleAssignEvent(String serviceUnit, ServiceUnitStateData data) {
+ deferGetOwnerRequest(serviceUnit);
+ if (isTargetBroker(data.broker())) {
+ ServiceUnitStateData next = new ServiceUnitStateData(
+ isTransferCommand(data) ? Released : Owned, data.broker(), data.sourceBroker());
+ pubAsync(serviceUnit, next)
+ .whenComplete((__, e) -> log(e, serviceUnit, data, next));
+ }
+ }
+
+ private void handleReleaseEvent(String serviceUnit, ServiceUnitStateData data) {
+
+ if (isTargetBroker(data.sourceBroker())) {
+ ServiceUnitStateData next = new ServiceUnitStateData(Owned, data.broker(), data.sourceBroker());
+ // TODO: when close, pass message to clients to connect to the new broker
+ closeServiceUnit(serviceUnit)
+ .thenCompose(__ -> pubAsync(serviceUnit, next))
+ .whenComplete((__, e) -> log(e, serviceUnit, data, next));
+ }
+ }
+
+ private void handleSplitEvent(String serviceUnit, ServiceUnitStateData data) {
+ if (isTargetBroker(data.broker())) {
+ splitServiceUnit(serviceUnit)
+ .thenCompose(__ -> tombstoneAsync(serviceUnit))
+ .whenComplete((__, e) -> log(e, serviceUnit, data, null));
+ }
+ }
+
+ private void handleTombstoneEvent(String serviceUnit) {
+ closeServiceUnit(serviceUnit)
+ .thenAccept(__ -> {
+ var request = getOwnerRequests.remove(serviceUnit);
+ if (request != null) {
+ request.completeExceptionally(new IllegalStateException("The ownership has been unloaded. "
+ + "No owner is found for serviceUnit: " + serviceUnit));
+ }
+ })
+ .whenComplete((__, e) -> log(e, serviceUnit, null, null));
+ }
+
+ private CompletableFuture<MessageId> pubAsync(String serviceUnit, ServiceUnitStateData data) {
+ CompletableFuture<MessageId> future = new CompletableFuture<>();
+ try {
+ outstandingPubMessages.acquire();
Review Comment:
blocking in a method that returns a CompletableFuture is usually a code smell and it leads to performance problems.
BUT At the moment I am not aware of a better way to do this in Pulsar codebase
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,779 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ private static final Schema<ServiceUnitStateData> SCHEMA = Schema.JSON(ServiceUnitStateData.class);
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private String lookupServiceAddress;
+ private LeaderElectionService leaderElectionService;
+ // TODO: define BrokerRegistry
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private Semaphore outstandingPubMessages;
+
+ private Semaphore outstandingCleanupTombstoneMessages;
+ private ScheduledFuture<?> cleanupTasks;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+
+ private ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ ServiceConfiguration conf = pulsar.getConfiguration();
+ this.lookupServiceAddress = pulsar.getAdvertisedAddress() + ":"
+ + (conf.getWebServicePort().isPresent() ? conf.getWebServicePort().get()
+ : conf.getWebServicePortTls().get());
+ this.outstandingPubMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder()
+ .build();
+ this.cleanupJobs =
+ ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder()
+ .build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ }
+
+ public void start() throws PulsarServerException {
+
+ try {
+ if (leaderElectionService != null) {
+ leaderElectionService.close();
+ log.debug("Closed the channel leader election service.");
+ }
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} was elected as the leader."
+ + "Current channel leader is {}",
+ lookupServiceAddress,
+ leaderElectionService.getCurrentLeader());
+ } else {
+ if (leaderElectionService != null) {
+ log.debug("This broker:{} is a follower. "
+ + "Current channel leader is {}",
+ lookupServiceAddress,
+ leaderElectionService.getCurrentLeader());
+ }
+ }
+ });
+ leaderElectionService.start();
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(SCHEMA)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(SCHEMA)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ // TODO: schedule cleanupTasks by brokerRegistry
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+
+ public void close() throws PulsarServerException {
+ try {
+ if (leaderElectionService != null) {
+ leaderElectionService.close();
+ leaderElectionService = null;
+ log.debug("Successfully closed the channel leader election service.");
+ }
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public void scheduleCompaction() throws PulsarServerException {
+ try {
+ Long threshold = pulsar.getAdminClient().topicPolicies()
+ .getCompactionThreshold(TOPIC);
+ if (threshold == null || threshold == 0) {
+ pulsar.getAdminClient().topicPolicies()
+ .setCompactionThreshold(TOPIC, COMPACTION_THRESHOLD);
+ log.info("Scheduled compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ } else {
+ log.info("Already set compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ }
+ } catch (PulsarAdminException e) {
+ throw new PulsarServerException("Failed to schedule compaction.", e);
+ }
+ }
+
+ public String getChannelOwner() {
+ CompletableFuture<Optional<LeaderBroker>> future = leaderElectionService.readCurrentLeader();
+ if (!future.isDone()) {
+ return null;
+ }
+ Optional<LeaderBroker> leader = future.join();
+ if (leader.isEmpty()) {
+ return null;
+ }
+ //expecting http://broker-xyz:port
+ String broker = leader.get().getServiceUrl();
+ broker = broker.substring(broker.lastIndexOf('/') + 1);
+ return broker;
+ }
+
+ public boolean isChannelOwner() {
+ return isTargetBroker(getChannelOwner());
+ }
+
+ public CompletableFuture<String> getOwnerAsync(String serviceUnit) {
+
+ ServiceUnitStateData data = tableview.get(serviceUnit);
+ if (data == null) {
+ return null;
+ }
+ switch (data.state()) {
+ case Owned, Splitting -> {
+ return CompletableFuture.completedFuture(data.broker());
+ }
+ case Assigned, Released -> {
+ return deferGetOwnerRequest(serviceUnit);
+ }
+ default -> {
+ return null;
+ }
+ }
+ }
+
+ public CompletableFuture<String> publishAssignEventAsync(String serviceUnit, String broker) {
+ CompletableFuture<String> getOwnerRequest = deferGetOwnerRequest(serviceUnit);
+ pubAsync(serviceUnit, new ServiceUnitStateData(Assigned, broker))
+ .whenComplete((__, ex) -> {
+ if (ex != null) {
+ getOwnerRequests.remove(serviceUnit);
+ if (!getOwnerRequest.isCompletedExceptionally()) {
+ getOwnerRequest.completeExceptionally(ex);
+ }
+ }
+ });
+
+ return getOwnerRequest;
+ }
+
+ public CompletableFuture<Void> publishUnloadEventAsync(Unload unload) {
+ String serviceUnit = unload.serviceUnit();
+ if (isTransferCommand(unload)) {
+ ServiceUnitStateData next = new ServiceUnitStateData(Assigned,
+ unload.destBroker().get(), unload.sourceBroker());
+ return pubAsync(serviceUnit, next).thenAccept(__ -> {
+ });
+ }
+ return tombstoneAsync(serviceUnit).thenAccept(__ -> {
+ });
+ }
+
+ public CompletableFuture<Void> publishSplitEventAsync(Split split) {
+ String serviceUnit = split.serviceUnit();
+ ServiceUnitStateData data = tableview.get(serviceUnit);
+ ServiceUnitStateData next = new ServiceUnitStateData(Splitting, data.broker());
+ return pubAsync(serviceUnit, next).thenAccept(__ -> {
+ });
+ }
+
+ private void handle(String serviceUnit, ServiceUnitStateData data) {
+ if (log.isDebugEnabled()) {
+ log.info("{} received a handle request for serviceUnit:{}, data:{}",
+ lookupServiceAddress, serviceUnit, data);
+ }
+
+ if (data == null) {
+ handleTombstoneEvent(serviceUnit);
+ return;
+ }
+
+ // TODO : Add state validation
+ switch (data.state()) {
+ case Owned -> handleOwnEvent(serviceUnit, data);
+ case Assigned -> handleAssignEvent(serviceUnit, data);
+ case Released -> handleReleaseEvent(serviceUnit, data);
+ case Splitting -> handleSplitEvent(serviceUnit, data);
+ default -> throw new IllegalStateException("Failed to handle channel data:" + data);
+ }
+ }
+
+ private static boolean isTransferCommand(ServiceUnitStateData data) {
+ if (data == null) {
+ return false;
+ }
+ return StringUtils.isNotEmpty(data.sourceBroker());
+ }
+
+ private static boolean isTransferCommand(Unload data) {
+ return data.destBroker().isPresent();
+ }
+
+ private static String getLogEventTag(ServiceUnitStateData data) {
+ return data == null ? "Free" :
+ isTransferCommand(data) ? "Transfer:" + data.state() : data.state().toString();
+ }
+
+ private void log(Throwable e, String serviceUnit, ServiceUnitStateData data, ServiceUnitStateData next) {
+ if (e == null) {
+ if (log.isDebugEnabled() || isTransferCommand(data)) {
+ log.info("{} handled {} event for serviceUnit:{}, cur:{}, next:{}",
+ lookupServiceAddress, getLogEventTag(data), serviceUnit,
+ data == null ? "" : data,
+ next == null ? "" : next);
+ }
+ } else {
+ log.error("{} failed to handle {} event for serviceUnit:{}, cur:{}, next:{}",
+ lookupServiceAddress, getLogEventTag(data), serviceUnit,
+ data == null ? "" : data,
+ next == null ? "" : next,
+ e);
+ }
+ }
+
+ private void handleOwnEvent(String serviceUnit, ServiceUnitStateData data) {
+ var getOwnerRequest = getOwnerRequests.remove(serviceUnit);
+ if (getOwnerRequest != null) {
+ getOwnerRequest.complete(data.broker());
+ }
+ if (isTargetBroker(data.broker())) {
+ log(null, serviceUnit, data, null);
+ }
+
+ }
+
+ private void handleAssignEvent(String serviceUnit, ServiceUnitStateData data) {
+ deferGetOwnerRequest(serviceUnit);
+ if (isTargetBroker(data.broker())) {
+ ServiceUnitStateData next = new ServiceUnitStateData(
+ isTransferCommand(data) ? Released : Owned, data.broker(), data.sourceBroker());
+ pubAsync(serviceUnit, next)
+ .whenComplete((__, e) -> log(e, serviceUnit, data, next));
+ }
+ }
+
+ private void handleReleaseEvent(String serviceUnit, ServiceUnitStateData data) {
+
+ if (isTargetBroker(data.sourceBroker())) {
+ ServiceUnitStateData next = new ServiceUnitStateData(Owned, data.broker(), data.sourceBroker());
+ // TODO: when close, pass message to clients to connect to the new broker
+ closeServiceUnit(serviceUnit)
+ .thenCompose(__ -> pubAsync(serviceUnit, next))
+ .whenComplete((__, e) -> log(e, serviceUnit, data, next));
+ }
+ }
+
+ private void handleSplitEvent(String serviceUnit, ServiceUnitStateData data) {
+ if (isTargetBroker(data.broker())) {
+ splitServiceUnit(serviceUnit)
+ .thenCompose(__ -> tombstoneAsync(serviceUnit))
+ .whenComplete((__, e) -> log(e, serviceUnit, data, null));
+ }
+ }
+
+ private void handleTombstoneEvent(String serviceUnit) {
+ closeServiceUnit(serviceUnit)
+ .thenAccept(__ -> {
+ var request = getOwnerRequests.remove(serviceUnit);
+ if (request != null) {
+ request.completeExceptionally(new IllegalStateException("The ownership has been unloaded. "
+ + "No owner is found for serviceUnit: " + serviceUnit));
+ }
+ })
+ .whenComplete((__, e) -> log(e, serviceUnit, null, null));
+ }
+
+ private CompletableFuture<MessageId> pubAsync(String serviceUnit, ServiceUnitStateData data) {
+ CompletableFuture<MessageId> future = new CompletableFuture<>();
+ try {
+ outstandingPubMessages.acquire();
+ } catch (InterruptedException e) {
+ log.warn("Interrupted while acquiring semaphore to publish a message for serviceUnit:{}, data:{}",
+ serviceUnit, data);
+ future.completeExceptionally(e);
+ return future;
+ }
+ producer.newMessage()
+ .key(serviceUnit)
+ .value(data)
+ .sendAsync()
+ .whenComplete((messageId, e) -> {
+ outstandingPubMessages.release();
+ if (e != null) {
+ future.completeExceptionally(e);
+ } else {
+ future.complete(messageId);
+ }
+ });
+ return future;
+ }
+
+ private CompletableFuture<MessageId> tombstoneAsync(String serviceUnit) {
+ return pubAsync(serviceUnit, null);
+ }
+
+ private boolean isTargetBroker(String broker) {
+ if (broker == null) {
+ return false;
+ }
+ // TODO: remove broker port from the input broker
Review Comment:
we should not remove the port, otherwise you won't be able to execute two brokers on the same host (I do it frequently with Pulsar standalone)
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,779 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ private static final Schema<ServiceUnitStateData> SCHEMA = Schema.JSON(ServiceUnitStateData.class);
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private String lookupServiceAddress;
+ private LeaderElectionService leaderElectionService;
+ // TODO: define BrokerRegistry
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private Semaphore outstandingPubMessages;
+
+ private Semaphore outstandingCleanupTombstoneMessages;
+ private ScheduledFuture<?> cleanupTasks;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+
+ private ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ ServiceConfiguration conf = pulsar.getConfiguration();
+ this.lookupServiceAddress = pulsar.getAdvertisedAddress() + ":"
+ + (conf.getWebServicePort().isPresent() ? conf.getWebServicePort().get()
+ : conf.getWebServicePortTls().get());
+ this.outstandingPubMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder()
+ .build();
+ this.cleanupJobs =
+ ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder()
+ .build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ }
+
+ public void start() throws PulsarServerException {
+
+ try {
+ if (leaderElectionService != null) {
+ leaderElectionService.close();
+ log.debug("Closed the channel leader election service.");
+ }
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} was elected as the leader."
+ + "Current channel leader is {}",
+ lookupServiceAddress,
+ leaderElectionService.getCurrentLeader());
+ } else {
+ if (leaderElectionService != null) {
+ log.debug("This broker:{} is a follower. "
+ + "Current channel leader is {}",
+ lookupServiceAddress,
+ leaderElectionService.getCurrentLeader());
+ }
+ }
+ });
+ leaderElectionService.start();
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(SCHEMA)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(SCHEMA)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ // TODO: schedule cleanupTasks by brokerRegistry
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+
+ public void close() throws PulsarServerException {
+ try {
+ if (leaderElectionService != null) {
+ leaderElectionService.close();
+ leaderElectionService = null;
+ log.debug("Successfully closed the channel leader election service.");
+ }
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public void scheduleCompaction() throws PulsarServerException {
+ try {
+ Long threshold = pulsar.getAdminClient().topicPolicies()
+ .getCompactionThreshold(TOPIC);
+ if (threshold == null || threshold == 0) {
+ pulsar.getAdminClient().topicPolicies()
+ .setCompactionThreshold(TOPIC, COMPACTION_THRESHOLD);
+ log.info("Scheduled compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ } else {
+ log.info("Already set compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ }
+ } catch (PulsarAdminException e) {
+ throw new PulsarServerException("Failed to schedule compaction.", e);
+ }
+ }
+
+ public String getChannelOwner() {
+ CompletableFuture<Optional<LeaderBroker>> future = leaderElectionService.readCurrentLeader();
+ if (!future.isDone()) {
+ return null;
+ }
+ Optional<LeaderBroker> leader = future.join();
+ if (leader.isEmpty()) {
+ return null;
+ }
+ //expecting http://broker-xyz:port
+ String broker = leader.get().getServiceUrl();
+ broker = broker.substring(broker.lastIndexOf('/') + 1);
Review Comment:
we should have some utility method to do this, maybe it is out of the scope of this PS
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,779 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ private static final Schema<ServiceUnitStateData> SCHEMA = Schema.JSON(ServiceUnitStateData.class);
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private String lookupServiceAddress;
+ private LeaderElectionService leaderElectionService;
+ // TODO: define BrokerRegistry
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private Semaphore outstandingPubMessages;
+
+ private Semaphore outstandingCleanupTombstoneMessages;
+ private ScheduledFuture<?> cleanupTasks;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+
+ private ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ ServiceConfiguration conf = pulsar.getConfiguration();
+ this.lookupServiceAddress = pulsar.getAdvertisedAddress() + ":"
+ + (conf.getWebServicePort().isPresent() ? conf.getWebServicePort().get()
+ : conf.getWebServicePortTls().get());
+ this.outstandingPubMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder()
+ .build();
+ this.cleanupJobs =
+ ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder()
+ .build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ }
+
+ public void start() throws PulsarServerException {
+
+ try {
+ if (leaderElectionService != null) {
+ leaderElectionService.close();
+ log.debug("Closed the channel leader election service.");
+ }
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} was elected as the leader."
+ + "Current channel leader is {}",
+ lookupServiceAddress,
+ leaderElectionService.getCurrentLeader());
+ } else {
+ if (leaderElectionService != null) {
+ log.debug("This broker:{} is a follower. "
+ + "Current channel leader is {}",
+ lookupServiceAddress,
+ leaderElectionService.getCurrentLeader());
+ }
+ }
+ });
+ leaderElectionService.start();
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(SCHEMA)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(SCHEMA)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ // TODO: schedule cleanupTasks by brokerRegistry
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+
+ public void close() throws PulsarServerException {
+ try {
+ if (leaderElectionService != null) {
+ leaderElectionService.close();
+ leaderElectionService = null;
+ log.debug("Successfully closed the channel leader election service.");
+ }
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public void scheduleCompaction() throws PulsarServerException {
+ try {
+ Long threshold = pulsar.getAdminClient().topicPolicies()
+ .getCompactionThreshold(TOPIC);
+ if (threshold == null || threshold == 0) {
+ pulsar.getAdminClient().topicPolicies()
+ .setCompactionThreshold(TOPIC, COMPACTION_THRESHOLD);
+ log.info("Scheduled compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ } else {
+ log.info("Already set compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ }
+ } catch (PulsarAdminException e) {
+ throw new PulsarServerException("Failed to schedule compaction.", e);
+ }
+ }
+
+ public String getChannelOwner() {
+ CompletableFuture<Optional<LeaderBroker>> future = leaderElectionService.readCurrentLeader();
+ if (!future.isDone()) {
+ return null;
+ }
+ Optional<LeaderBroker> leader = future.join();
+ if (leader.isEmpty()) {
+ return null;
+ }
+ //expecting http://broker-xyz:port
+ String broker = leader.get().getServiceUrl();
+ broker = broker.substring(broker.lastIndexOf('/') + 1);
+ return broker;
+ }
+
+ public boolean isChannelOwner() {
+ return isTargetBroker(getChannelOwner());
+ }
+
+ public CompletableFuture<String> getOwnerAsync(String serviceUnit) {
+
+ ServiceUnitStateData data = tableview.get(serviceUnit);
+ if (data == null) {
+ return null;
+ }
+ switch (data.state()) {
+ case Owned, Splitting -> {
+ return CompletableFuture.completedFuture(data.broker());
+ }
+ case Assigned, Released -> {
+ return deferGetOwnerRequest(serviceUnit);
+ }
+ default -> {
+ return null;
+ }
+ }
+ }
+
+ public CompletableFuture<String> publishAssignEventAsync(String serviceUnit, String broker) {
+ CompletableFuture<String> getOwnerRequest = deferGetOwnerRequest(serviceUnit);
+ pubAsync(serviceUnit, new ServiceUnitStateData(Assigned, broker))
+ .whenComplete((__, ex) -> {
+ if (ex != null) {
+ getOwnerRequests.remove(serviceUnit);
+ if (!getOwnerRequest.isCompletedExceptionally()) {
+ getOwnerRequest.completeExceptionally(ex);
+ }
+ }
+ });
+
+ return getOwnerRequest;
+ }
+
+ public CompletableFuture<Void> publishUnloadEventAsync(Unload unload) {
+ String serviceUnit = unload.serviceUnit();
+ if (isTransferCommand(unload)) {
+ ServiceUnitStateData next = new ServiceUnitStateData(Assigned,
+ unload.destBroker().get(), unload.sourceBroker());
+ return pubAsync(serviceUnit, next).thenAccept(__ -> {
+ });
+ }
+ return tombstoneAsync(serviceUnit).thenAccept(__ -> {
+ });
+ }
+
+ public CompletableFuture<Void> publishSplitEventAsync(Split split) {
+ String serviceUnit = split.serviceUnit();
+ ServiceUnitStateData data = tableview.get(serviceUnit);
+ ServiceUnitStateData next = new ServiceUnitStateData(Splitting, data.broker());
+ return pubAsync(serviceUnit, next).thenAccept(__ -> {
+ });
+ }
+
+ private void handle(String serviceUnit, ServiceUnitStateData data) {
+ if (log.isDebugEnabled()) {
+ log.info("{} received a handle request for serviceUnit:{}, data:{}",
+ lookupServiceAddress, serviceUnit, data);
+ }
+
+ if (data == null) {
+ handleTombstoneEvent(serviceUnit);
+ return;
+ }
+
+ // TODO : Add state validation
+ switch (data.state()) {
+ case Owned -> handleOwnEvent(serviceUnit, data);
+ case Assigned -> handleAssignEvent(serviceUnit, data);
+ case Released -> handleReleaseEvent(serviceUnit, data);
+ case Splitting -> handleSplitEvent(serviceUnit, data);
+ default -> throw new IllegalStateException("Failed to handle channel data:" + data);
+ }
+ }
+
+ private static boolean isTransferCommand(ServiceUnitStateData data) {
+ if (data == null) {
+ return false;
+ }
+ return StringUtils.isNotEmpty(data.sourceBroker());
+ }
+
+ private static boolean isTransferCommand(Unload data) {
+ return data.destBroker().isPresent();
+ }
+
+ private static String getLogEventTag(ServiceUnitStateData data) {
+ return data == null ? "Free" :
+ isTransferCommand(data) ? "Transfer:" + data.state() : data.state().toString();
+ }
+
+ private void log(Throwable e, String serviceUnit, ServiceUnitStateData data, ServiceUnitStateData next) {
Review Comment:
nit: static ?
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,779 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ private static final Schema<ServiceUnitStateData> SCHEMA = Schema.JSON(ServiceUnitStateData.class);
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private String lookupServiceAddress;
+ private LeaderElectionService leaderElectionService;
+ // TODO: define BrokerRegistry
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private Semaphore outstandingPubMessages;
+
+ private Semaphore outstandingCleanupTombstoneMessages;
+ private ScheduledFuture<?> cleanupTasks;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+
+ private ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ ServiceConfiguration conf = pulsar.getConfiguration();
+ this.lookupServiceAddress = pulsar.getAdvertisedAddress() + ":"
+ + (conf.getWebServicePort().isPresent() ? conf.getWebServicePort().get()
+ : conf.getWebServicePortTls().get());
+ this.outstandingPubMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder()
+ .build();
+ this.cleanupJobs =
+ ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder()
+ .build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ }
+
+ public void start() throws PulsarServerException {
+
+ try {
+ if (leaderElectionService != null) {
+ leaderElectionService.close();
+ log.debug("Closed the channel leader election service.");
+ }
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} was elected as the leader."
+ + "Current channel leader is {}",
+ lookupServiceAddress,
+ leaderElectionService.getCurrentLeader());
+ } else {
+ if (leaderElectionService != null) {
+ log.debug("This broker:{} is a follower. "
+ + "Current channel leader is {}",
+ lookupServiceAddress,
+ leaderElectionService.getCurrentLeader());
+ }
+ }
+ });
+ leaderElectionService.start();
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(SCHEMA)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(SCHEMA)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ // TODO: schedule cleanupTasks by brokerRegistry
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+
+ public void close() throws PulsarServerException {
+ try {
+ if (leaderElectionService != null) {
+ leaderElectionService.close();
+ leaderElectionService = null;
+ log.debug("Successfully closed the channel leader election service.");
+ }
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public void scheduleCompaction() throws PulsarServerException {
+ try {
+ Long threshold = pulsar.getAdminClient().topicPolicies()
+ .getCompactionThreshold(TOPIC);
+ if (threshold == null || threshold == 0) {
+ pulsar.getAdminClient().topicPolicies()
+ .setCompactionThreshold(TOPIC, COMPACTION_THRESHOLD);
+ log.info("Scheduled compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ } else {
+ log.info("Already set compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ }
+ } catch (PulsarAdminException e) {
+ throw new PulsarServerException("Failed to schedule compaction.", e);
+ }
+ }
+
+ public String getChannelOwner() {
+ CompletableFuture<Optional<LeaderBroker>> future = leaderElectionService.readCurrentLeader();
Review Comment:
I think that we are not handling concurrent access on "leaderElectionService"
in "start" we re-assign the value to the variable.
I would make it final or properly handle concurrent access
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] shibd commented on a diff in pull request #18489: [improve][broker] PIP-192 Added ServiceUnitStateChannelImpl
Posted by GitBox <gi...@apache.org>.
shibd commented on code in PR #18489:
URL: https://github.com/apache/pulsar/pull/18489#discussion_r1027177087
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,761 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ private static final Schema<ServiceUnitStateData> SCHEMA = Schema.JSON(ServiceUnitStateData.class);
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private final String lookupServiceAddress;
+ // TODO: define BrokerRegistry
+ private final Semaphore outstandingCleanupTombstoneMessages;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+ private LeaderElectionService leaderElectionService;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private ScheduledFuture<?> cleanupTasks;
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+ private volatile boolean isActive;
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.isActive = false;
+ this.pulsar = pulsar;
+ ServiceConfiguration conf = pulsar.getConfiguration();
+ this.lookupServiceAddress = pulsar.getLookupServiceAddress();
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder().build();
+ this.cleanupJobs = ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder().build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ }
+
+ public synchronized void start() throws PulsarServerException {
+ try {
+ if (leaderElectionService != null) {
+ leaderElectionService.close();
+ log.debug("Closed the channel leader election service.");
+ }
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} is the leader now.", lookupServiceAddress);
+ // TODO: schedule monitorOwnerships by brokerRegistry
+ } else {
+ log.debug("This broker:{} is a follower now.", lookupServiceAddress);
+ // TODO: cancel scheduled monitorOwnerships if any
+ }
+ });
+ leaderElectionService.start();
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(SCHEMA)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(SCHEMA)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ isActive = true;
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public synchronized void close() throws PulsarServerException {
+ isActive = false;
+ try {
+ if (leaderElectionService != null) {
+ leaderElectionService.close();
+ leaderElectionService = null;
+ log.debug("Successfully closed the channel leader election service.");
+ }
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ private void validateChannel() {
+ if (!isActive) {
+ throw new IllegalStateException("The channel has not been started.");
+ }
+ }
+
+ public synchronized void scheduleCompaction() throws PulsarServerException {
+ try {
+ Long threshold = pulsar.getAdminClient().topicPolicies()
+ .getCompactionThreshold(TOPIC);
+ if (threshold == null || threshold == 0) {
+ pulsar.getAdminClient().topicPolicies()
+ .setCompactionThreshold(TOPIC, COMPACTION_THRESHOLD);
+ log.info("Scheduled compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ } else {
+ log.info("Already set compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ }
+ } catch (PulsarAdminException e) {
+ throw new PulsarServerException("Failed to schedule compaction.", e);
+ }
+ }
+
+ public String getChannelOwner() {
+ validateChannel();
+ CompletableFuture<Optional<LeaderBroker>> future = leaderElectionService.readCurrentLeader();
+ if (!future.isDone()) {
+ return null;
+ }
+ Optional<LeaderBroker> leader = future.join();
+ if (leader.isEmpty()) {
+ return null;
+ }
+ //expecting http://broker-xyz:port
+ // TODO: discard this protocol prefix removal by a util func that returns lookupServiceAddress(serviceUrl)
+ String broker = leader.get().getServiceUrl();
+ broker = broker.substring(broker.lastIndexOf('/') + 1);
+ return broker;
+ }
+
+ public boolean isChannelOwner() {
+ return isTargetBroker(getChannelOwner());
+ }
+
+ public CompletableFuture<String> getOwnerAsync(String serviceUnit) {
+ validateChannel();
+ ServiceUnitStateData data = tableview.get(serviceUnit);
+ if (data == null) {
+ return null;
+ }
+ switch (data.state()) {
+ case Owned, Splitting -> {
+ return CompletableFuture.completedFuture(data.broker());
+ }
+ case Assigned, Released -> {
+ return deferGetOwnerRequest(serviceUnit);
+ }
+ default -> {
+ return null;
+ }
+ }
+ }
+
+ public CompletableFuture<String> publishAssignEventAsync(String serviceUnit, String broker) {
+ CompletableFuture<String> getOwnerRequest = deferGetOwnerRequest(serviceUnit);
+ pubAsync(serviceUnit, new ServiceUnitStateData(Assigned, broker))
+ .whenComplete((__, ex) -> {
+ if (ex != null) {
+ getOwnerRequests.remove(serviceUnit);
+ if (!getOwnerRequest.isCompletedExceptionally()) {
+ getOwnerRequest.completeExceptionally(ex);
+ }
+ }
+ });
+
+ return getOwnerRequest;
+ }
+
+ public CompletableFuture<Void> publishUnloadEventAsync(Unload unload) {
+ String serviceUnit = unload.serviceUnit();
+ if (isTransferCommand(unload)) {
+ ServiceUnitStateData next = new ServiceUnitStateData(Assigned,
Review Comment:
Here should publish `Free` status?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] shibd commented on a diff in pull request #18489: [improve][broker] PIP-192 Added ServiceUnitStateChannelImpl
Posted by GitBox <gi...@apache.org>.
shibd commented on code in PR #18489:
URL: https://github.com/apache/pulsar/pull/18489#discussion_r1027178748
##########
pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelTest.java:
##########
@@ -0,0 +1,772 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.ConnectionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.Reconnected;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.greaterThanOrEqualTo;
+import static org.hamcrest.Matchers.lessThanOrEqualTo;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.assertNull;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.eq;
+import static org.mockito.Mockito.doReturn;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.spy;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.testng.Assert.assertFalse;
+import static org.testng.Assert.assertTrue;
+import static org.testng.AssertJUnit.assertNotNull;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import org.apache.commons.lang.StringUtils;
+import org.apache.commons.lang3.reflect.FieldUtils;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.auth.MockedPulsarServiceBaseTest;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.TypedMessageBuilder;
+import org.apache.pulsar.client.impl.TableViewImpl;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+import org.awaitility.Awaitility;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.Test;
+
+@Test(groups = "broker")
+public class ServiceUnitStateChannelTest extends MockedPulsarServiceBaseTest {
+
+ private PulsarService pulsar1;
+ private PulsarService pulsar2;
+ private ServiceUnitStateChannel channel1;
+ private ServiceUnitStateChannel channel2;
+ private String lookupServiceAddress1;
+ private String lookupServiceAddress2;
+ private String bundle;
+
+ private String bundle1;
+ private String bundle2;
+
+ @BeforeClass
+ @Override
+ protected void setup() throws Exception {
+ conf.setAllowAutoTopicCreation(true);
+ super.internalSetup(conf);
+
+ admin.tenants().createTenant("pulsar", createDefaultTenantInfo());
+ admin.namespaces().createNamespace("pulsar/system");
+ admin.tenants().createTenant("public", createDefaultTenantInfo());
+ admin.namespaces().createNamespace("public/default");
+
+ pulsar1 = pulsar;
+ pulsar2 = startBrokerWithoutAuthorization(getDefaultConf());
+ channel1 = new ServiceUnitStateChannelImpl(pulsar1);
+ channel1.start();
+ channel2 = new ServiceUnitStateChannelImpl(pulsar2);
+ channel2.start();
+ lookupServiceAddress1 = (String)
+ FieldUtils.readDeclaredField(channel1, "lookupServiceAddress", true);
+ lookupServiceAddress2 = (String)
+ FieldUtils.readDeclaredField(channel2, "lookupServiceAddress", true);
+
+ bundle = String.format("%s/%s", "public/default", "0x00000000_0xffffffff");
+ bundle1 = String.format("%s/%s", "public/default", "0x00000000_0xfffffff0");
+ bundle2 = String.format("%s/%s", "public/default", "0xfffffff0_0xffffffff");
+ }
+
+ @BeforeMethod
+ protected void initTableViews() throws Exception {
+ cleanTableView(channel1, bundle);
+ cleanTableView(channel2, bundle);
+ }
+
+
+ @AfterClass
+ @Override
+ protected void cleanup() throws Exception {
+ channel1.close();
+ channel2.close();
+ pulsar1 = null;
+ pulsar2.close();
+ super.internalCleanup();
+ }
+
+ @Test(priority = 0)
+ public void channelOwnerTest() throws Exception {
+ var channelOwner1 = channel1.getChannelOwner();
+ var channelOwner2 = channel2.getChannelOwner();
+ assertEquals(channelOwner1, channelOwner2);
+ LeaderElectionService leaderElectionService1 = (LeaderElectionService) FieldUtils.readDeclaredField(
+ channel1, "leaderElectionService", true);
+ leaderElectionService1.close();
+ waitUntilNewChannelOwner(channel2, channelOwner1);
+ leaderElectionService1.start();
+ waitUntilNewChannelOwner(channel1, channelOwner1);
+
+ var newChannelOwner1 = channel1.getChannelOwner();
+ var newChannelOwner2 = channel2.getChannelOwner();
+
+ assertEquals(newChannelOwner1, newChannelOwner2);
+ assertNotEquals(channelOwner1, newChannelOwner1);
+
+ if (newChannelOwner1.equals(lookupServiceAddress1)) {
+ assertTrue(channel1.isChannelOwner());
+ assertFalse(channel2.isChannelOwner());
+ } else {
+ assertFalse(channel1.isChannelOwner());
+ assertTrue(channel2.isChannelOwner());
+ }
+ }
+
+ @Test(priority = 0)
+ public void channelValidationTest() throws ExecutionException, InterruptedException {
+ var channel = new ServiceUnitStateChannelImpl(pulsar);
+
+ int errorCnt = validateChannelStart(channel);
+ assertEquals(6, errorCnt);
+ ExecutorService executor = Executors.newSingleThreadExecutor();
+ Future startFuture = executor.submit(() -> {
+ try {
+ channel.start();
+ } catch (PulsarServerException e) {
+ throw new RuntimeException(e);
+ }
+ });
+ errorCnt = validateChannelStart(channel);
+ startFuture.get();
+ assertTrue(errorCnt > 0);
+ Future closeFuture = executor.submit(()->{
+ try {
+ channel.close();
+ } catch (PulsarServerException e) {
+ throw new RuntimeException(e);
+ }
+ });
+ errorCnt = validateChannelStart(channel);
+ closeFuture.get();
+ assertTrue(errorCnt > 0);
+ errorCnt = validateChannelStart(channel);
+ assertEquals(6, errorCnt);
+ }
+
+ private int validateChannelStart(ServiceUnitStateChannelImpl channel) {
+ int errorCnt = 0;
+ try {
+ channel.isChannelOwner();
+ } catch (IllegalStateException e) {
+ errorCnt++;
+ }
+ try {
+ channel.getChannelOwner();
+ } catch (IllegalStateException e) {
+ errorCnt++;
+ }
+ try {
+ channel.getOwnerAsync(bundle);
+ } catch (IllegalStateException e) {
+ errorCnt++;
+ }
+ try {
+ channel.publishAssignEventAsync(bundle, lookupServiceAddress1);
+ } catch (IllegalStateException e) {
+ errorCnt++;
+ }
+ try {
+ channel.publishUnloadEventAsync(
+ new Unload(lookupServiceAddress1, bundle, Optional.of(lookupServiceAddress2)));
+ } catch (IllegalStateException e) {
+ errorCnt++;
+ }
+ try {
+ channel.publishSplitEventAsync(new Split(bundle, lookupServiceAddress1, Map.of()));
+ } catch (IllegalStateException e) {
+ errorCnt++;
+ }
+ return errorCnt;
+ }
+
+ @Test(priority = 1)
+ public void compactionScheduleTest() throws PulsarServerException {
+
+ channel1.scheduleCompaction();
+
+ Awaitility.await()
+ .pollInterval(200, TimeUnit.MILLISECONDS)
+ .atMost(5, TimeUnit.SECONDS)
+ .untilAsserted(() -> { // wait until true
+ try {
+ var threshold = admin.topicPolicies()
+ .getCompactionThreshold(ServiceUnitStateChannelImpl.TOPIC, false).longValue();
+ assertEquals(5 * 1024 * 1024, threshold);
+ } catch (Exception e) {
+ ;
+ }
+
+ });
+ }
+
+ @Test(priority = 2)
+ public void assignmentTest()
+ throws ExecutionException, InterruptedException, IllegalAccessException, TimeoutException {
+ var getOwnerRequests1 = spy(getOwnerRequests((channel1)));
+ var getOwnerRequests2 = spy(getOwnerRequests((channel2)));
+
+ var owner1 = channel1.getOwnerAsync(bundle);
+ var owner2 = channel2.getOwnerAsync(bundle);
+
+ assertNull(owner1);
+ assertNull(owner2);
+
+ var assigned1 = channel1.publishAssignEventAsync(bundle, lookupServiceAddress1);
+ var assigned2 = channel2.publishAssignEventAsync(bundle, lookupServiceAddress2);
+ assertNotNull(assigned1);
+ assertNotNull(assigned2);
+ waitUntilOwnerChanges(channel2, bundle, null);
Review Comment:
```suggestion
waitUntilOwnerChanges(channel1, bundle, null);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] heesung-sn commented on a diff in pull request #18489: [improve][broker] PIP-192 Added ServiceUnitStateChannelImpl
Posted by GitBox <gi...@apache.org>.
heesung-sn commented on code in PR #18489:
URL: https://github.com/apache/pulsar/pull/18489#discussion_r1027225022
##########
pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelTest.java:
##########
@@ -0,0 +1,772 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.ConnectionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.Reconnected;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.greaterThanOrEqualTo;
+import static org.hamcrest.Matchers.lessThanOrEqualTo;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.assertNull;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.eq;
+import static org.mockito.Mockito.doReturn;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.spy;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.testng.Assert.assertFalse;
+import static org.testng.Assert.assertTrue;
+import static org.testng.AssertJUnit.assertNotNull;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import org.apache.commons.lang.StringUtils;
+import org.apache.commons.lang3.reflect.FieldUtils;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.auth.MockedPulsarServiceBaseTest;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.TypedMessageBuilder;
+import org.apache.pulsar.client.impl.TableViewImpl;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+import org.awaitility.Awaitility;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.Test;
+
+@Test(groups = "broker")
+public class ServiceUnitStateChannelTest extends MockedPulsarServiceBaseTest {
+
+ private PulsarService pulsar1;
+ private PulsarService pulsar2;
+ private ServiceUnitStateChannel channel1;
+ private ServiceUnitStateChannel channel2;
+ private String lookupServiceAddress1;
+ private String lookupServiceAddress2;
+ private String bundle;
+
+ private String bundle1;
+ private String bundle2;
+
+ @BeforeClass
+ @Override
+ protected void setup() throws Exception {
+ conf.setAllowAutoTopicCreation(true);
+ super.internalSetup(conf);
+
+ admin.tenants().createTenant("pulsar", createDefaultTenantInfo());
+ admin.namespaces().createNamespace("pulsar/system");
+ admin.tenants().createTenant("public", createDefaultTenantInfo());
+ admin.namespaces().createNamespace("public/default");
+
+ pulsar1 = pulsar;
+ pulsar2 = startBrokerWithoutAuthorization(getDefaultConf());
+ channel1 = new ServiceUnitStateChannelImpl(pulsar1);
+ channel1.start();
+ channel2 = new ServiceUnitStateChannelImpl(pulsar2);
+ channel2.start();
+ lookupServiceAddress1 = (String)
+ FieldUtils.readDeclaredField(channel1, "lookupServiceAddress", true);
+ lookupServiceAddress2 = (String)
+ FieldUtils.readDeclaredField(channel2, "lookupServiceAddress", true);
+
+ bundle = String.format("%s/%s", "public/default", "0x00000000_0xffffffff");
+ bundle1 = String.format("%s/%s", "public/default", "0x00000000_0xfffffff0");
+ bundle2 = String.format("%s/%s", "public/default", "0xfffffff0_0xffffffff");
+ }
+
+ @BeforeMethod
+ protected void initTableViews() throws Exception {
+ cleanTableView(channel1, bundle);
+ cleanTableView(channel2, bundle);
+ }
+
+
+ @AfterClass
+ @Override
+ protected void cleanup() throws Exception {
+ channel1.close();
+ channel2.close();
+ pulsar1 = null;
+ pulsar2.close();
+ super.internalCleanup();
+ }
+
+ @Test(priority = 0)
+ public void channelOwnerTest() throws Exception {
+ var channelOwner1 = channel1.getChannelOwner();
+ var channelOwner2 = channel2.getChannelOwner();
+ assertEquals(channelOwner1, channelOwner2);
+ LeaderElectionService leaderElectionService1 = (LeaderElectionService) FieldUtils.readDeclaredField(
+ channel1, "leaderElectionService", true);
+ leaderElectionService1.close();
+ waitUntilNewChannelOwner(channel2, channelOwner1);
+ leaderElectionService1.start();
+ waitUntilNewChannelOwner(channel1, channelOwner1);
+
+ var newChannelOwner1 = channel1.getChannelOwner();
+ var newChannelOwner2 = channel2.getChannelOwner();
+
+ assertEquals(newChannelOwner1, newChannelOwner2);
+ assertNotEquals(channelOwner1, newChannelOwner1);
+
+ if (newChannelOwner1.equals(lookupServiceAddress1)) {
+ assertTrue(channel1.isChannelOwner());
+ assertFalse(channel2.isChannelOwner());
+ } else {
+ assertFalse(channel1.isChannelOwner());
+ assertTrue(channel2.isChannelOwner());
+ }
+ }
+
+ @Test(priority = 0)
+ public void channelValidationTest() throws ExecutionException, InterruptedException {
+ var channel = new ServiceUnitStateChannelImpl(pulsar);
+
+ int errorCnt = validateChannelStart(channel);
+ assertEquals(6, errorCnt);
+ ExecutorService executor = Executors.newSingleThreadExecutor();
+ Future startFuture = executor.submit(() -> {
+ try {
+ channel.start();
+ } catch (PulsarServerException e) {
+ throw new RuntimeException(e);
+ }
+ });
+ errorCnt = validateChannelStart(channel);
+ startFuture.get();
+ assertTrue(errorCnt > 0);
+ Future closeFuture = executor.submit(()->{
+ try {
+ channel.close();
+ } catch (PulsarServerException e) {
+ throw new RuntimeException(e);
+ }
+ });
+ errorCnt = validateChannelStart(channel);
+ closeFuture.get();
+ assertTrue(errorCnt > 0);
+ errorCnt = validateChannelStart(channel);
+ assertEquals(6, errorCnt);
+ }
+
+ private int validateChannelStart(ServiceUnitStateChannelImpl channel) {
+ int errorCnt = 0;
+ try {
+ channel.isChannelOwner();
+ } catch (IllegalStateException e) {
+ errorCnt++;
+ }
+ try {
+ channel.getChannelOwner();
+ } catch (IllegalStateException e) {
+ errorCnt++;
+ }
+ try {
+ channel.getOwnerAsync(bundle);
+ } catch (IllegalStateException e) {
+ errorCnt++;
+ }
+ try {
+ channel.publishAssignEventAsync(bundle, lookupServiceAddress1);
+ } catch (IllegalStateException e) {
+ errorCnt++;
+ }
+ try {
+ channel.publishUnloadEventAsync(
+ new Unload(lookupServiceAddress1, bundle, Optional.of(lookupServiceAddress2)));
+ } catch (IllegalStateException e) {
+ errorCnt++;
+ }
+ try {
+ channel.publishSplitEventAsync(new Split(bundle, lookupServiceAddress1, Map.of()));
+ } catch (IllegalStateException e) {
+ errorCnt++;
+ }
+ return errorCnt;
+ }
+
+ @Test(priority = 1)
+ public void compactionScheduleTest() throws PulsarServerException {
+
+ channel1.scheduleCompaction();
+
+ Awaitility.await()
+ .pollInterval(200, TimeUnit.MILLISECONDS)
+ .atMost(5, TimeUnit.SECONDS)
+ .untilAsserted(() -> { // wait until true
+ try {
+ var threshold = admin.topicPolicies()
+ .getCompactionThreshold(ServiceUnitStateChannelImpl.TOPIC, false).longValue();
+ assertEquals(5 * 1024 * 1024, threshold);
+ } catch (Exception e) {
+ ;
+ }
+
+ });
+ }
+
+ @Test(priority = 2)
+ public void assignmentTest()
+ throws ExecutionException, InterruptedException, IllegalAccessException, TimeoutException {
+ var getOwnerRequests1 = spy(getOwnerRequests((channel1)));
+ var getOwnerRequests2 = spy(getOwnerRequests((channel2)));
+
+ var owner1 = channel1.getOwnerAsync(bundle);
+ var owner2 = channel2.getOwnerAsync(bundle);
+
+ assertNull(owner1);
+ assertNull(owner2);
+
+ var assigned1 = channel1.publishAssignEventAsync(bundle, lookupServiceAddress1);
+ var assigned2 = channel2.publishAssignEventAsync(bundle, lookupServiceAddress2);
+ assertNotNull(assigned1);
+ assertNotNull(assigned2);
+ waitUntilOwnerChanges(channel2, bundle, null);
Review Comment:
I think adding the `Command` layer is additional complexity and requires more lines of code. Also, the current approach does not need to store the additional `Command` info in the system topic(space optimized).
Regarding `ServiceUnitState.isValidTransition()`, the state validation(aka compaction) will happen in the tableview and topic compaction by `TopicCompactionStrategy`(which is in a separate PIP and PR here, https://github.com/apache/pulsar/issues/18099). This validation logic, ServiceUnitChannelCompactionStrategy will be added after this PR https://github.com/apache/pulsar/pull/18195 gets merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] heesung-sn commented on a diff in pull request #18489: [improve][broker] PIP-192 Added ServiceUnitStateChannelImpl
Posted by GitBox <gi...@apache.org>.
heesung-sn commented on code in PR #18489:
URL: https://github.com/apache/pulsar/pull/18489#discussion_r1034310124
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,778 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Free;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Closed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Constructed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.LeaderElectionServiceStarted;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Started;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private final Schema<ServiceUnitStateData> schema;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private final String lookupServiceAddress;
+ // TODO: define BrokerRegistry
+ private final Semaphore outstandingCleanupTombstoneMessages;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+ private final LeaderElectionService leaderElectionService;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private ScheduledFuture<?> cleanupTasks;
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+ private volatile ChannelState channelState;
+
+ enum ChannelState {
+ Closed(0),
+ Constructed(1),
+ LeaderElectionServiceStarted(2),
+ Started(3);
+
+ ChannelState(int id) {
+ this.id = id;
+ }
+ int id;
+ }
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ this.lookupServiceAddress = pulsar.getLookupServiceAddress();
+ this.schema = Schema.JSON(ServiceUnitStateData.class);
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder().build();
+ this.cleanupJobs = ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder().build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} is the leader now.", lookupServiceAddress);
+ // TODO: schedule monitorOwnerships by brokerRegistry
+ } else {
+ log.debug("This broker:{} is a follower now.", lookupServiceAddress);
+ // TODO: cancel scheduled monitorOwnerships if any
+ }
+ });
+ this.channelState = Constructed;
+ }
+
+ public synchronized void start() throws PulsarServerException {
Review Comment:
No. The channelOwner will own the system topic.(The leaderElectionServiceStarted channel state can return getChannelOwner api.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] heesung-sn commented on a diff in pull request #18489: [improve][broker] PIP-192 Added ServiceUnitStateChannelImpl
Posted by GitBox <gi...@apache.org>.
heesung-sn commented on code in PR #18489:
URL: https://github.com/apache/pulsar/pull/18489#discussion_r1034423639
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,778 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Free;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Closed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Constructed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.LeaderElectionServiceStarted;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Started;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private final Schema<ServiceUnitStateData> schema;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private final String lookupServiceAddress;
+ // TODO: define BrokerRegistry
+ private final Semaphore outstandingCleanupTombstoneMessages;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+ private final LeaderElectionService leaderElectionService;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private ScheduledFuture<?> cleanupTasks;
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+ private volatile ChannelState channelState;
+
+ enum ChannelState {
+ Closed(0),
+ Constructed(1),
+ LeaderElectionServiceStarted(2),
+ Started(3);
+
+ ChannelState(int id) {
+ this.id = id;
+ }
+ int id;
+ }
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ this.lookupServiceAddress = pulsar.getLookupServiceAddress();
+ this.schema = Schema.JSON(ServiceUnitStateData.class);
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder().build();
+ this.cleanupJobs = ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder().build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} is the leader now.", lookupServiceAddress);
+ // TODO: schedule monitorOwnerships by brokerRegistry
+ } else {
+ log.debug("This broker:{} is a follower now.", lookupServiceAddress);
+ // TODO: cancel scheduled monitorOwnerships if any
+ }
+ });
+ this.channelState = Constructed;
+ }
+
+ public synchronized void start() throws PulsarServerException {
+ validateChannelState(LeaderElectionServiceStarted, false);
+ try {
+ leaderElectionService.start();
+ this.channelState = LeaderElectionServiceStarted;
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(schema)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(schema)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ channelState = Started;
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public synchronized void close() throws PulsarServerException {
+ channelState = Closed;
+ try {
+ leaderElectionService.close();
+ log.debug("Successfully closed the channel leader election service.");
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ private void validateChannelState(ChannelState targetState, boolean checkLowerIds) {
+ int order = checkLowerIds ? -1 : 1;
+ if (Integer.compare(channelState.id, targetState.id) * order > 0) {
+ throw new IllegalStateException("Invalid channel state:" + channelState.name());
+ }
+ }
+
+ public synchronized void scheduleCompaction() throws PulsarServerException {
+ try {
+ Long threshold = pulsar.getAdminClient().topicPolicies()
+ .getCompactionThreshold(TOPIC);
+ if (threshold == null || threshold == 0) {
+ pulsar.getAdminClient().topicPolicies()
+ .setCompactionThreshold(TOPIC, COMPACTION_THRESHOLD);
+ log.info("Scheduled compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ } else {
+ log.info("Already set compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ }
+ } catch (PulsarAdminException e) {
+ throw new PulsarServerException("Failed to schedule compaction.", e);
+ }
+ }
+
+ public String getChannelOwner() {
+ validateChannelState(LeaderElectionServiceStarted, true);
+ CompletableFuture<Optional<LeaderBroker>> future = leaderElectionService.readCurrentLeader();
+ if (!future.isDone()) {
+ return null;
+ }
+ Optional<LeaderBroker> leader = future.join();
+ if (leader.isEmpty()) {
+ return null;
+ }
+ //expecting http://broker-xyz:port
+ // TODO: discard this protocol prefix removal by a util func that returns lookupServiceAddress(serviceUrl)
+ String broker = leader.get().getServiceUrl();
+ broker = broker.substring(broker.lastIndexOf('/') + 1);
+ return broker;
+ }
+
+ public boolean isChannelOwner() {
+ return isTargetBroker(getChannelOwner());
+ }
+
+ public CompletableFuture<String> getOwnerAsync(String serviceUnit) {
+ validateChannelState(Started, true);
+ ServiceUnitStateData data = tableview.get(serviceUnit);
+ if (data == null) {
+ return null;
+ }
+ switch (data.state()) {
+ case Owned, Splitting -> {
+ return CompletableFuture.completedFuture(data.broker());
+ }
+ case Assigned, Released -> {
+ return deferGetOwnerRequest(serviceUnit);
+ }
+ default -> {
+ return null;
+ }
+ }
+ }
+
+ public CompletableFuture<String> publishAssignEventAsync(String serviceUnit, String broker) {
+ CompletableFuture<String> getOwnerRequest = deferGetOwnerRequest(serviceUnit);
+ pubAsync(serviceUnit, new ServiceUnitStateData(Assigned, broker))
+ .whenComplete((__, ex) -> {
+ if (ex != null) {
+ getOwnerRequests.remove(serviceUnit);
+ if (!getOwnerRequest.isCompletedExceptionally()) {
+ getOwnerRequest.completeExceptionally(ex);
+ }
+ }
+ });
+
+ return getOwnerRequest;
+ }
+
+ public CompletableFuture<Void> publishUnloadEventAsync(Unload unload) {
+ String serviceUnit = unload.serviceUnit();
+ if (isTransferCommand(unload)) {
+ ServiceUnitStateData next = new ServiceUnitStateData(Assigned,
+ unload.destBroker().get(), unload.sourceBroker());
+ return pubAsync(serviceUnit, next).thenAccept(__ -> {
+ });
+ }
+ return tombstoneAsync(serviceUnit).thenAccept(__ -> {
+ });
+ }
+
+ public CompletableFuture<Void> publishSplitEventAsync(Split split) {
+ String serviceUnit = split.serviceUnit();
+ ServiceUnitStateData next = new ServiceUnitStateData(Splitting, split.sourceBroker());
+ return pubAsync(serviceUnit, next).thenAccept(__ -> {
+ });
+ }
+
+ private void handle(String serviceUnit, ServiceUnitStateData data) {
+ if (log.isDebugEnabled()) {
+ log.info("{} received a handle request for serviceUnit:{}, data:{}",
+ lookupServiceAddress, serviceUnit, data);
+ }
+
+ ServiceUnitState state = data == null ? Free : data.state();
+
+ // TODO : Add state validation in tableview by the compaction strategy
+ switch (state) {
+ case Owned -> handleOwnEvent(serviceUnit, data);
+ case Assigned -> handleAssignEvent(serviceUnit, data);
+ case Released -> handleReleaseEvent(serviceUnit, data);
+ case Splitting -> handleSplitEvent(serviceUnit, data);
+ case Free -> handleFreeEvent(serviceUnit);
+ default -> throw new IllegalStateException("Failed to handle channel data:" + data);
+ }
+ }
+
+ private static boolean isTransferCommand(ServiceUnitStateData data) {
+ if (data == null) {
+ return false;
+ }
+ return StringUtils.isNotEmpty(data.sourceBroker());
+ }
+
+ private static boolean isTransferCommand(Unload data) {
+ return data.destBroker().isPresent();
+ }
+
+ private static String getLogEventTag(ServiceUnitStateData data) {
+ return data == null ? "Free" :
+ isTransferCommand(data) ? "Transfer:" + data.state() : data.state().toString();
+ }
+
+ private void log(Throwable e, String serviceUnit, ServiceUnitStateData data, ServiceUnitStateData next) {
+ if (e == null) {
+ if (log.isDebugEnabled() || isTransferCommand(data)) {
+ log.info("{} handled {} event for serviceUnit:{}, cur:{}, next:{}",
+ lookupServiceAddress, getLogEventTag(data), serviceUnit,
+ data == null ? "" : data,
+ next == null ? "" : next);
+ }
+ } else {
+ log.error("{} failed to handle {} event for serviceUnit:{}, cur:{}, next:{}",
+ lookupServiceAddress, getLogEventTag(data), serviceUnit,
+ data == null ? "" : data,
+ next == null ? "" : next,
+ e);
+ }
+ }
+
+ private void handleOwnEvent(String serviceUnit, ServiceUnitStateData data) {
+ var getOwnerRequest = getOwnerRequests.remove(serviceUnit);
+ if (getOwnerRequest != null) {
+ getOwnerRequest.complete(data.broker());
+ }
+ if (isTargetBroker(data.broker())) {
+ log(null, serviceUnit, data, null);
+ }
+
+ }
+
+ private void handleAssignEvent(String serviceUnit, ServiceUnitStateData data) {
+ deferGetOwnerRequest(serviceUnit);
+ if (isTargetBroker(data.broker())) {
+ ServiceUnitStateData next = new ServiceUnitStateData(
+ isTransferCommand(data) ? Released : Owned, data.broker(), data.sourceBroker());
+ pubAsync(serviceUnit, next)
+ .whenComplete((__, e) -> log(e, serviceUnit, data, next));
+ }
+ }
+
+ private void handleReleaseEvent(String serviceUnit, ServiceUnitStateData data) {
+
+ if (isTargetBroker(data.sourceBroker())) {
+ ServiceUnitStateData next = new ServiceUnitStateData(Owned, data.broker(), data.sourceBroker());
+ // TODO: when close, pass message to clients to connect to the new broker
+ closeServiceUnit(serviceUnit)
+ .thenCompose(__ -> pubAsync(serviceUnit, next))
+ .whenComplete((__, e) -> log(e, serviceUnit, data, next));
+ }
+ }
+
+ private void handleSplitEvent(String serviceUnit, ServiceUnitStateData data) {
+ if (isTargetBroker(data.broker())) {
+ splitServiceUnit(serviceUnit)
+ .thenCompose(__ -> tombstoneAsync(serviceUnit))
+ .whenComplete((__, e) -> log(e, serviceUnit, data, null));
+ }
+ }
+
+ private void handleFreeEvent(String serviceUnit) {
+ closeServiceUnit(serviceUnit)
+ .thenAccept(__ -> {
+ var request = getOwnerRequests.remove(serviceUnit);
+ if (request != null) {
+ request.completeExceptionally(new IllegalStateException("The ownership has been unloaded. "
+ + "No owner is found for serviceUnit: " + serviceUnit));
+ }
+ })
+ .whenComplete((__, e) -> log(e, serviceUnit, null, null));
+ }
+
+ private CompletableFuture<MessageId> pubAsync(String serviceUnit, ServiceUnitStateData data) {
+ validateChannelState(Started, true);
+ CompletableFuture<MessageId> future = new CompletableFuture<>();
+ producer.newMessage()
+ .key(serviceUnit)
+ .value(data)
+ .sendAsync()
+ .whenComplete((messageId, e) -> {
+ if (e != null) {
+ log.error("Failed to publish the message: serviceUnit:{}, data:{}",
+ serviceUnit, data, e);
+ future.completeExceptionally(e);
+ } else {
+ future.complete(messageId);
+ }
+ });
+ return future;
+ }
+
+ private CompletableFuture<MessageId> tombstoneAsync(String serviceUnit) {
+ return pubAsync(serviceUnit, null);
+ }
+
+ private boolean isTargetBroker(String broker) {
+ if (broker == null) {
+ return false;
+ }
+ return broker.equals(lookupServiceAddress);
+ }
+
+ private NamespaceBundle getNamespaceBundle(String bundle) {
+ final String namespaceName = LoadManagerShared.getNamespaceNameFromBundleName(bundle);
+ final String bundleRange = LoadManagerShared.getBundleRangeFromBundleName(bundle);
+ return pulsar.getNamespaceService().getNamespaceBundleFactory().getBundle(namespaceName, bundleRange);
+ }
+
+ private CompletableFuture<String> deferGetOwnerRequest(String serviceUnit) {
+ return getOwnerRequests
+ .computeIfAbsent(serviceUnit, k -> {
+ CompletableFuture future = new CompletableFuture<>();
+ future.orTimeout(inFlightStateWaitingTimeInMillis, TimeUnit.MILLISECONDS)
+ .whenComplete((v, e) -> {
+ if (e != null) {
+ getOwnerRequests.remove(serviceUnit);
+ log.warn("Failed to getOwner for serviceUnit:{}",
+ serviceUnit, e);
+ future.completeExceptionally((Throwable) e);
+ }
+ }
+ );
+ return future;
+ });
+ }
+
+ private CompletableFuture<Integer> closeServiceUnit(String serviceUnit) {
+ long startTime = System.nanoTime();
+ MutableInt unloadedTopics = new MutableInt();
+ NamespaceBundle bundle = getNamespaceBundle(serviceUnit);
+ return pulsar.getBrokerService().unloadServiceUnit(
+ bundle,
+ false,
+ pulsar.getConfig().getNamespaceBundleUnloadingTimeoutMs(),
+ TimeUnit.MILLISECONDS)
+ .handle((numUnloadedTopics, ex) -> {
+ if (ex != null) {
+ // ignore topic-close failure to unload bundle
+ log.error("Failed to close topics under bundle:{}",
+ bundle.toString(), ex);
+ } else {
+ unloadedTopics.setValue(numUnloadedTopics);
+ }
+ // clean up topics that failed to unload from the broker ownership cache
+ pulsar.getBrokerService().cleanUnloadedTopicFromCache(bundle);
+ return numUnloadedTopics;
+ })
+ .whenComplete((ignored, ex) -> {
+ double unloadBundleTime = TimeUnit.NANOSECONDS
+ .toMillis((System.nanoTime() - startTime));
+ log.info("Unloading bundle:{} with {} topics completed in {} ms",
+ bundle, unloadedTopics, unloadBundleTime, ex);
Review Comment:
Although this part is from the original code, updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] heesung-sn commented on a diff in pull request #18489: [improve][broker] PIP-192 Added ServiceUnitStateChannelImpl
Posted by GitBox <gi...@apache.org>.
heesung-sn commented on code in PR #18489:
URL: https://github.com/apache/pulsar/pull/18489#discussion_r1027225039
##########
pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelTest.java:
##########
@@ -0,0 +1,772 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.ConnectionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.Reconnected;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.greaterThanOrEqualTo;
+import static org.hamcrest.Matchers.lessThanOrEqualTo;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.assertNull;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.eq;
+import static org.mockito.Mockito.doReturn;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.spy;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.testng.Assert.assertFalse;
+import static org.testng.Assert.assertTrue;
+import static org.testng.AssertJUnit.assertNotNull;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import org.apache.commons.lang.StringUtils;
+import org.apache.commons.lang3.reflect.FieldUtils;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.auth.MockedPulsarServiceBaseTest;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.TypedMessageBuilder;
+import org.apache.pulsar.client.impl.TableViewImpl;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+import org.awaitility.Awaitility;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.Test;
+
+@Test(groups = "broker")
+public class ServiceUnitStateChannelTest extends MockedPulsarServiceBaseTest {
+
+ private PulsarService pulsar1;
+ private PulsarService pulsar2;
+ private ServiceUnitStateChannel channel1;
+ private ServiceUnitStateChannel channel2;
+ private String lookupServiceAddress1;
+ private String lookupServiceAddress2;
+ private String bundle;
+
+ private String bundle1;
+ private String bundle2;
+
+ @BeforeClass
+ @Override
+ protected void setup() throws Exception {
+ conf.setAllowAutoTopicCreation(true);
+ super.internalSetup(conf);
+
+ admin.tenants().createTenant("pulsar", createDefaultTenantInfo());
+ admin.namespaces().createNamespace("pulsar/system");
+ admin.tenants().createTenant("public", createDefaultTenantInfo());
+ admin.namespaces().createNamespace("public/default");
+
+ pulsar1 = pulsar;
+ pulsar2 = startBrokerWithoutAuthorization(getDefaultConf());
+ channel1 = new ServiceUnitStateChannelImpl(pulsar1);
+ channel1.start();
+ channel2 = new ServiceUnitStateChannelImpl(pulsar2);
+ channel2.start();
+ lookupServiceAddress1 = (String)
+ FieldUtils.readDeclaredField(channel1, "lookupServiceAddress", true);
+ lookupServiceAddress2 = (String)
+ FieldUtils.readDeclaredField(channel2, "lookupServiceAddress", true);
+
+ bundle = String.format("%s/%s", "public/default", "0x00000000_0xffffffff");
+ bundle1 = String.format("%s/%s", "public/default", "0x00000000_0xfffffff0");
+ bundle2 = String.format("%s/%s", "public/default", "0xfffffff0_0xffffffff");
+ }
+
+ @BeforeMethod
+ protected void initTableViews() throws Exception {
+ cleanTableView(channel1, bundle);
+ cleanTableView(channel2, bundle);
+ }
+
+
+ @AfterClass
+ @Override
+ protected void cleanup() throws Exception {
+ channel1.close();
+ channel2.close();
+ pulsar1 = null;
+ pulsar2.close();
+ super.internalCleanup();
+ }
+
+ @Test(priority = 0)
+ public void channelOwnerTest() throws Exception {
+ var channelOwner1 = channel1.getChannelOwner();
+ var channelOwner2 = channel2.getChannelOwner();
+ assertEquals(channelOwner1, channelOwner2);
+ LeaderElectionService leaderElectionService1 = (LeaderElectionService) FieldUtils.readDeclaredField(
+ channel1, "leaderElectionService", true);
+ leaderElectionService1.close();
+ waitUntilNewChannelOwner(channel2, channelOwner1);
+ leaderElectionService1.start();
+ waitUntilNewChannelOwner(channel1, channelOwner1);
+
+ var newChannelOwner1 = channel1.getChannelOwner();
+ var newChannelOwner2 = channel2.getChannelOwner();
+
+ assertEquals(newChannelOwner1, newChannelOwner2);
+ assertNotEquals(channelOwner1, newChannelOwner1);
+
+ if (newChannelOwner1.equals(lookupServiceAddress1)) {
+ assertTrue(channel1.isChannelOwner());
+ assertFalse(channel2.isChannelOwner());
+ } else {
+ assertFalse(channel1.isChannelOwner());
+ assertTrue(channel2.isChannelOwner());
+ }
+ }
+
+ @Test(priority = 0)
+ public void channelValidationTest() throws ExecutionException, InterruptedException {
+ var channel = new ServiceUnitStateChannelImpl(pulsar);
+
+ int errorCnt = validateChannelStart(channel);
+ assertEquals(6, errorCnt);
+ ExecutorService executor = Executors.newSingleThreadExecutor();
+ Future startFuture = executor.submit(() -> {
+ try {
+ channel.start();
+ } catch (PulsarServerException e) {
+ throw new RuntimeException(e);
+ }
+ });
+ errorCnt = validateChannelStart(channel);
+ startFuture.get();
+ assertTrue(errorCnt > 0);
+ Future closeFuture = executor.submit(()->{
+ try {
+ channel.close();
+ } catch (PulsarServerException e) {
+ throw new RuntimeException(e);
+ }
+ });
+ errorCnt = validateChannelStart(channel);
+ closeFuture.get();
+ assertTrue(errorCnt > 0);
+ errorCnt = validateChannelStart(channel);
+ assertEquals(6, errorCnt);
+ }
+
+ private int validateChannelStart(ServiceUnitStateChannelImpl channel) {
+ int errorCnt = 0;
+ try {
+ channel.isChannelOwner();
+ } catch (IllegalStateException e) {
+ errorCnt++;
+ }
+ try {
+ channel.getChannelOwner();
+ } catch (IllegalStateException e) {
+ errorCnt++;
+ }
+ try {
+ channel.getOwnerAsync(bundle);
+ } catch (IllegalStateException e) {
+ errorCnt++;
+ }
+ try {
+ channel.publishAssignEventAsync(bundle, lookupServiceAddress1);
+ } catch (IllegalStateException e) {
+ errorCnt++;
+ }
+ try {
+ channel.publishUnloadEventAsync(
+ new Unload(lookupServiceAddress1, bundle, Optional.of(lookupServiceAddress2)));
+ } catch (IllegalStateException e) {
+ errorCnt++;
+ }
+ try {
+ channel.publishSplitEventAsync(new Split(bundle, lookupServiceAddress1, Map.of()));
+ } catch (IllegalStateException e) {
+ errorCnt++;
+ }
+ return errorCnt;
+ }
+
+ @Test(priority = 1)
+ public void compactionScheduleTest() throws PulsarServerException {
+
+ channel1.scheduleCompaction();
+
+ Awaitility.await()
+ .pollInterval(200, TimeUnit.MILLISECONDS)
+ .atMost(5, TimeUnit.SECONDS)
+ .untilAsserted(() -> { // wait until true
+ try {
+ var threshold = admin.topicPolicies()
+ .getCompactionThreshold(ServiceUnitStateChannelImpl.TOPIC, false).longValue();
+ assertEquals(5 * 1024 * 1024, threshold);
+ } catch (Exception e) {
+ ;
+ }
+
+ });
+ }
+
+ @Test(priority = 2)
+ public void assignmentTest()
+ throws ExecutionException, InterruptedException, IllegalAccessException, TimeoutException {
+ var getOwnerRequests1 = spy(getOwnerRequests((channel1)));
+ var getOwnerRequests2 = spy(getOwnerRequests((channel2)));
+
+ var owner1 = channel1.getOwnerAsync(bundle);
+ var owner2 = channel2.getOwnerAsync(bundle);
+
+ assertNull(owner1);
+ assertNull(owner2);
+
+ var assigned1 = channel1.publishAssignEventAsync(bundle, lookupServiceAddress1);
+ var assigned2 = channel2.publishAssignEventAsync(bundle, lookupServiceAddress2);
+ assertNotNull(assigned1);
+ assertNotNull(assigned2);
+ waitUntilOwnerChanges(channel2, bundle, null);
Review Comment:
Thanks for the `waitUntilOwnerChanges(channel1, bundle, null);` catch.
updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] heesung-sn commented on a diff in pull request #18489: [improve][broker] PIP-192 Added ServiceUnitStateChannelImpl
Posted by GitBox <gi...@apache.org>.
heesung-sn commented on code in PR #18489:
URL: https://github.com/apache/pulsar/pull/18489#discussion_r1027224326
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,779 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ private static final Schema<ServiceUnitStateData> SCHEMA = Schema.JSON(ServiceUnitStateData.class);
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private String lookupServiceAddress;
+ private LeaderElectionService leaderElectionService;
+ // TODO: define BrokerRegistry
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private Semaphore outstandingPubMessages;
+
+ private Semaphore outstandingCleanupTombstoneMessages;
+ private ScheduledFuture<?> cleanupTasks;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+
+ private ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ ServiceConfiguration conf = pulsar.getConfiguration();
+ this.lookupServiceAddress = pulsar.getAdvertisedAddress() + ":"
+ + (conf.getWebServicePort().isPresent() ? conf.getWebServicePort().get()
+ : conf.getWebServicePortTls().get());
+ this.outstandingPubMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder()
+ .build();
+ this.cleanupJobs =
+ ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder()
+ .build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ }
+
+ public void start() throws PulsarServerException {
+
+ try {
+ if (leaderElectionService != null) {
+ leaderElectionService.close();
+ log.debug("Closed the channel leader election service.");
+ }
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} was elected as the leader."
+ + "Current channel leader is {}",
+ lookupServiceAddress,
+ leaderElectionService.getCurrentLeader());
+ } else {
+ if (leaderElectionService != null) {
+ log.debug("This broker:{} is a follower. "
+ + "Current channel leader is {}",
+ lookupServiceAddress,
+ leaderElectionService.getCurrentLeader());
+ }
+ }
+ });
+ leaderElectionService.start();
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(SCHEMA)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(SCHEMA)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ // TODO: schedule cleanupTasks by brokerRegistry
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+
+ public void close() throws PulsarServerException {
+ try {
+ if (leaderElectionService != null) {
+ leaderElectionService.close();
+ leaderElectionService = null;
+ log.debug("Successfully closed the channel leader election service.");
+ }
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public void scheduleCompaction() throws PulsarServerException {
+ try {
+ Long threshold = pulsar.getAdminClient().topicPolicies()
+ .getCompactionThreshold(TOPIC);
+ if (threshold == null || threshold == 0) {
+ pulsar.getAdminClient().topicPolicies()
+ .setCompactionThreshold(TOPIC, COMPACTION_THRESHOLD);
+ log.info("Scheduled compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ } else {
+ log.info("Already set compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ }
+ } catch (PulsarAdminException e) {
+ throw new PulsarServerException("Failed to schedule compaction.", e);
+ }
+ }
+
+ public String getChannelOwner() {
+ CompletableFuture<Optional<LeaderBroker>> future = leaderElectionService.readCurrentLeader();
Review Comment:
In fact, we need further the channel state breakdown (`boolean isActive` is not enough because `getChannelOwner()` should be accessible while creating the `producer` and `tableview` on the system topic -- the topic owner lookup will call `getChannelOwner()`.
I added the `enum StateChannel` to differentiate the channel states.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] heesung-sn commented on a diff in pull request #18489: [improve][broker] PIP-192 Added ServiceUnitStateChannelImpl
Posted by GitBox <gi...@apache.org>.
heesung-sn commented on code in PR #18489:
URL: https://github.com/apache/pulsar/pull/18489#discussion_r1027241928
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,779 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ private static final Schema<ServiceUnitStateData> SCHEMA = Schema.JSON(ServiceUnitStateData.class);
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private String lookupServiceAddress;
+ private LeaderElectionService leaderElectionService;
+ // TODO: define BrokerRegistry
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private Semaphore outstandingPubMessages;
+
+ private Semaphore outstandingCleanupTombstoneMessages;
+ private ScheduledFuture<?> cleanupTasks;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+
+ private ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ ServiceConfiguration conf = pulsar.getConfiguration();
+ this.lookupServiceAddress = pulsar.getAdvertisedAddress() + ":"
+ + (conf.getWebServicePort().isPresent() ? conf.getWebServicePort().get()
+ : conf.getWebServicePortTls().get());
+ this.outstandingPubMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder()
+ .build();
+ this.cleanupJobs =
+ ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder()
+ .build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ }
+
+ public void start() throws PulsarServerException {
+
+ try {
+ if (leaderElectionService != null) {
+ leaderElectionService.close();
+ log.debug("Closed the channel leader election service.");
+ }
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} was elected as the leader."
+ + "Current channel leader is {}",
+ lookupServiceAddress,
+ leaderElectionService.getCurrentLeader());
+ } else {
+ if (leaderElectionService != null) {
+ log.debug("This broker:{} is a follower. "
+ + "Current channel leader is {}",
+ lookupServiceAddress,
+ leaderElectionService.getCurrentLeader());
+ }
+ }
+ });
+ leaderElectionService.start();
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(SCHEMA)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(SCHEMA)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ // TODO: schedule cleanupTasks by brokerRegistry
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+
+ public void close() throws PulsarServerException {
+ try {
+ if (leaderElectionService != null) {
+ leaderElectionService.close();
+ leaderElectionService = null;
+ log.debug("Successfully closed the channel leader election service.");
+ }
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public void scheduleCompaction() throws PulsarServerException {
+ try {
+ Long threshold = pulsar.getAdminClient().topicPolicies()
+ .getCompactionThreshold(TOPIC);
+ if (threshold == null || threshold == 0) {
+ pulsar.getAdminClient().topicPolicies()
+ .setCompactionThreshold(TOPIC, COMPACTION_THRESHOLD);
+ log.info("Scheduled compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ } else {
+ log.info("Already set compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ }
+ } catch (PulsarAdminException e) {
+ throw new PulsarServerException("Failed to schedule compaction.", e);
+ }
+ }
+
+ public String getChannelOwner() {
+ CompletableFuture<Optional<LeaderBroker>> future = leaderElectionService.readCurrentLeader();
Review Comment:
About the concurrent access to the member variables, still 'tableview' and 'producer' are not final.
Could you explain why we need the synchronized access to the 'leaderElectionService' instance?
(Don't we also need to worry about the synchronized access to tableview and producer?)
As long as it's non-null, shouldn't it be ok whether the return(leader) is from the old or new instance of 'leaderElectionService'?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] heesung-sn commented on a diff in pull request #18489: [improve][broker] PIP-192 Added ServiceUnitStateChannelImpl
Posted by GitBox <gi...@apache.org>.
heesung-sn commented on code in PR #18489:
URL: https://github.com/apache/pulsar/pull/18489#discussion_r1027259946
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,779 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ private static final Schema<ServiceUnitStateData> SCHEMA = Schema.JSON(ServiceUnitStateData.class);
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private String lookupServiceAddress;
+ private LeaderElectionService leaderElectionService;
+ // TODO: define BrokerRegistry
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private Semaphore outstandingPubMessages;
+
+ private Semaphore outstandingCleanupTombstoneMessages;
+ private ScheduledFuture<?> cleanupTasks;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+
+ private ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ ServiceConfiguration conf = pulsar.getConfiguration();
+ this.lookupServiceAddress = pulsar.getAdvertisedAddress() + ":"
+ + (conf.getWebServicePort().isPresent() ? conf.getWebServicePort().get()
+ : conf.getWebServicePortTls().get());
+ this.outstandingPubMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder()
+ .build();
+ this.cleanupJobs =
+ ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder()
+ .build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ }
+
+ public void start() throws PulsarServerException {
+
+ try {
+ if (leaderElectionService != null) {
+ leaderElectionService.close();
+ log.debug("Closed the channel leader election service.");
+ }
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} was elected as the leader."
+ + "Current channel leader is {}",
+ lookupServiceAddress,
+ leaderElectionService.getCurrentLeader());
+ } else {
+ if (leaderElectionService != null) {
+ log.debug("This broker:{} is a follower. "
+ + "Current channel leader is {}",
+ lookupServiceAddress,
+ leaderElectionService.getCurrentLeader());
+ }
+ }
+ });
+ leaderElectionService.start();
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(SCHEMA)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(SCHEMA)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ // TODO: schedule cleanupTasks by brokerRegistry
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+
+ public void close() throws PulsarServerException {
+ try {
+ if (leaderElectionService != null) {
+ leaderElectionService.close();
+ leaderElectionService = null;
+ log.debug("Successfully closed the channel leader election service.");
+ }
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public void scheduleCompaction() throws PulsarServerException {
+ try {
+ Long threshold = pulsar.getAdminClient().topicPolicies()
+ .getCompactionThreshold(TOPIC);
+ if (threshold == null || threshold == 0) {
+ pulsar.getAdminClient().topicPolicies()
+ .setCompactionThreshold(TOPIC, COMPACTION_THRESHOLD);
+ log.info("Scheduled compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ } else {
+ log.info("Already set compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ }
+ } catch (PulsarAdminException e) {
+ throw new PulsarServerException("Failed to schedule compaction.", e);
+ }
+ }
+
+ public String getChannelOwner() {
+ CompletableFuture<Optional<LeaderBroker>> future = leaderElectionService.readCurrentLeader();
Review Comment:
Aren't the current state channel validation and synchronized start/close func enough? The invalid channel state can throw exception when the getter funcs are called during start() and close(). AFAIK, the previous setter instructions (producer and tableview setter) should be piggybacked by the volatile channelState=Started flush(by the JMM happen-before piggyback)
If this is not sufficient, can you explain why? And do you suggest forcefully synchronized access like 'synchronized getProducer()/setProducer() and synchronized getTableView()/setTableView()?
We could use ReadWriteLock to optimize this, but isn't it an overkill?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] heesung-sn commented on pull request #18489: [improve][broker] PIP-192 Added ServiceUnitStateChannelImpl
Posted by GitBox <gi...@apache.org>.
heesung-sn commented on PR #18489:
URL: https://github.com/apache/pulsar/pull/18489#issuecomment-1317844321
this PR implements https://github.com/apache/pulsar/pull/18079
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] heesung-sn commented on a diff in pull request #18489: [improve][broker] PIP-192 Added ServiceUnitStateChannelImpl
Posted by GitBox <gi...@apache.org>.
heesung-sn commented on code in PR #18489:
URL: https://github.com/apache/pulsar/pull/18489#discussion_r1027241928
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,779 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ private static final Schema<ServiceUnitStateData> SCHEMA = Schema.JSON(ServiceUnitStateData.class);
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private String lookupServiceAddress;
+ private LeaderElectionService leaderElectionService;
+ // TODO: define BrokerRegistry
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private Semaphore outstandingPubMessages;
+
+ private Semaphore outstandingCleanupTombstoneMessages;
+ private ScheduledFuture<?> cleanupTasks;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+
+ private ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ ServiceConfiguration conf = pulsar.getConfiguration();
+ this.lookupServiceAddress = pulsar.getAdvertisedAddress() + ":"
+ + (conf.getWebServicePort().isPresent() ? conf.getWebServicePort().get()
+ : conf.getWebServicePortTls().get());
+ this.outstandingPubMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder()
+ .build();
+ this.cleanupJobs =
+ ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder()
+ .build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ }
+
+ public void start() throws PulsarServerException {
+
+ try {
+ if (leaderElectionService != null) {
+ leaderElectionService.close();
+ log.debug("Closed the channel leader election service.");
+ }
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} was elected as the leader."
+ + "Current channel leader is {}",
+ lookupServiceAddress,
+ leaderElectionService.getCurrentLeader());
+ } else {
+ if (leaderElectionService != null) {
+ log.debug("This broker:{} is a follower. "
+ + "Current channel leader is {}",
+ lookupServiceAddress,
+ leaderElectionService.getCurrentLeader());
+ }
+ }
+ });
+ leaderElectionService.start();
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(SCHEMA)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(SCHEMA)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ // TODO: schedule cleanupTasks by brokerRegistry
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+
+ public void close() throws PulsarServerException {
+ try {
+ if (leaderElectionService != null) {
+ leaderElectionService.close();
+ leaderElectionService = null;
+ log.debug("Successfully closed the channel leader election service.");
+ }
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public void scheduleCompaction() throws PulsarServerException {
+ try {
+ Long threshold = pulsar.getAdminClient().topicPolicies()
+ .getCompactionThreshold(TOPIC);
+ if (threshold == null || threshold == 0) {
+ pulsar.getAdminClient().topicPolicies()
+ .setCompactionThreshold(TOPIC, COMPACTION_THRESHOLD);
+ log.info("Scheduled compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ } else {
+ log.info("Already set compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ }
+ } catch (PulsarAdminException e) {
+ throw new PulsarServerException("Failed to schedule compaction.", e);
+ }
+ }
+
+ public String getChannelOwner() {
+ CompletableFuture<Optional<LeaderBroker>> future = leaderElectionService.readCurrentLeader();
Review Comment:
About the concurrent access to the member variables, still 'tableview' and 'producer' are not final.
Could you explain why we need the synchronized access to the 'leaderElectionService' instance(but care less(no synchronized access) about tableview and producer?
As long as it's non-null, shouldn't it be ok whether the return(leader) is from the old or new instance of 'leaderElectionService'?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codecov-commenter commented on pull request #18489: [improve][broker] PIP-192 Added ServiceUnitStateChannelImpl
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #18489:
URL: https://github.com/apache/pulsar/pull/18489#issuecomment-1325088900
# [Codecov](https://codecov.io/gh/apache/pulsar/pull/18489?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#18489](https://codecov.io/gh/apache/pulsar/pull/18489?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (07d6022) into [master](https://codecov.io/gh/apache/pulsar/commit/79750231051db849350e3d35dd3706320466acac?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7975023) will **decrease** coverage by `8.49%`.
> The diff coverage is `59.37%`.
[](https://codecov.io/gh/apache/pulsar/pull/18489?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #18489 +/- ##
============================================
- Coverage 45.62% 37.12% -8.50%
+ Complexity 10075 1969 -8106
============================================
Files 697 209 -488
Lines 68024 14393 -53631
Branches 7293 1569 -5724
============================================
- Hits 31033 5344 -25689
+ Misses 33413 8466 -24947
+ Partials 3578 583 -2995
```
| Flag | Coverage Δ | |
|---|---|---|
| unittests | `37.12% <59.37%> (-8.50%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pulsar/pull/18489?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...nt/impl/PulsarClientImplementationBindingImpl.java](https://codecov.io/gh/apache/pulsar/pull/18489/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL1B1bHNhckNsaWVudEltcGxlbWVudGF0aW9uQmluZGluZ0ltcGwuamF2YQ==) | `72.41% <ø> (-0.47%)` | :arrow_down: |
| [...ar/client/impl/conf/ConsumerConfigurationData.java](https://codecov.io/gh/apache/pulsar/pull/18489/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL2NvbmYvQ29uc3VtZXJDb25maWd1cmF0aW9uRGF0YS5qYXZh) | `92.55% <ø> (+2.12%)` | :arrow_up: |
| [...he/pulsar/client/impl/TypedMessageBuilderImpl.java](https://codecov.io/gh/apache/pulsar/pull/18489/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL1R5cGVkTWVzc2FnZUJ1aWxkZXJJbXBsLmphdmE=) | `30.76% <59.37%> (+3.56%)` | :arrow_up: |
| [...va/org/apache/pulsar/client/impl/ConsumerBase.java](https://codecov.io/gh/apache/pulsar/pull/18489/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL0NvbnN1bWVyQmFzZS5qYXZh) | `21.93% <0.00%> (-0.02%)` | :arrow_down: |
| [...rg/apache/pulsar/broker/admin/v3/Transactions.java](https://codecov.io/gh/apache/pulsar/pull/18489/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9hZG1pbi92My9UcmFuc2FjdGlvbnMuamF2YQ==) | | |
| [.../apache/pulsar/broker/admin/v1/ResourceQuotas.java](https://codecov.io/gh/apache/pulsar/pull/18489/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9hZG1pbi92MS9SZXNvdXJjZVF1b3Rhcy5qYXZh) | | |
| [...g/apache/bookkeeper/mledger/impl/PositionImpl.java](https://codecov.io/gh/apache/pulsar/pull/18489/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-bWFuYWdlZC1sZWRnZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2Jvb2trZWVwZXIvbWxlZGdlci9pbXBsL1Bvc2l0aW9uSW1wbC5qYXZh) | | |
| [.../service/SystemTopicBasedTopicPoliciesService.java](https://codecov.io/gh/apache/pulsar/pull/18489/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9zZXJ2aWNlL1N5c3RlbVRvcGljQmFzZWRUb3BpY1BvbGljaWVzU2VydmljZS5qYXZh) | | |
| [...apache/pulsar/proxy/server/ProxyConfiguration.java](https://codecov.io/gh/apache/pulsar/pull/18489/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLXByb3h5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9wdWxzYXIvcHJveHkvc2VydmVyL1Byb3h5Q29uZmlndXJhdGlvbi5qYXZh) | | |
| [...broker/delayed/InMemoryDelayedDeliveryTracker.java](https://codecov.io/gh/apache/pulsar/pull/18489/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9kZWxheWVkL0luTWVtb3J5RGVsYXllZERlbGl2ZXJ5VHJhY2tlci5qYXZh) | | |
| ... and [491 more](https://codecov.io/gh/apache/pulsar/pull/18489/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on a diff in pull request #18489: [improve][broker] PIP-192 Added ServiceUnitStateChannelImpl
Posted by GitBox <gi...@apache.org>.
eolivelli commented on code in PR #18489:
URL: https://github.com/apache/pulsar/pull/18489#discussion_r1027228879
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,776 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Closed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Constructed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.LeaderElectionServiceStarted;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Started;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ private static final Schema<ServiceUnitStateData> SCHEMA = Schema.JSON(ServiceUnitStateData.class);
Review Comment:
We should not cache Schema objects in "static" variables.
Schema is stateful and it is injected with a reference to the PulsarClient. Maybe this is not the case for JSON and for the current version, but it is better to not do it.
Please move this to a instance field
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,779 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ private static final Schema<ServiceUnitStateData> SCHEMA = Schema.JSON(ServiceUnitStateData.class);
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private String lookupServiceAddress;
+ private LeaderElectionService leaderElectionService;
+ // TODO: define BrokerRegistry
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private Semaphore outstandingPubMessages;
+
+ private Semaphore outstandingCleanupTombstoneMessages;
+ private ScheduledFuture<?> cleanupTasks;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+
+ private ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ ServiceConfiguration conf = pulsar.getConfiguration();
+ this.lookupServiceAddress = pulsar.getAdvertisedAddress() + ":"
+ + (conf.getWebServicePort().isPresent() ? conf.getWebServicePort().get()
+ : conf.getWebServicePortTls().get());
+ this.outstandingPubMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder()
+ .build();
+ this.cleanupJobs =
+ ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder()
+ .build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ }
+
+ public void start() throws PulsarServerException {
+
+ try {
+ if (leaderElectionService != null) {
+ leaderElectionService.close();
+ log.debug("Closed the channel leader election service.");
+ }
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} was elected as the leader."
+ + "Current channel leader is {}",
+ lookupServiceAddress,
+ leaderElectionService.getCurrentLeader());
+ } else {
+ if (leaderElectionService != null) {
+ log.debug("This broker:{} is a follower. "
+ + "Current channel leader is {}",
+ lookupServiceAddress,
+ leaderElectionService.getCurrentLeader());
+ }
+ }
+ });
+ leaderElectionService.start();
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(SCHEMA)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(SCHEMA)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ // TODO: schedule cleanupTasks by brokerRegistry
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+
+ public void close() throws PulsarServerException {
+ try {
+ if (leaderElectionService != null) {
+ leaderElectionService.close();
+ leaderElectionService = null;
+ log.debug("Successfully closed the channel leader election service.");
+ }
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public void scheduleCompaction() throws PulsarServerException {
+ try {
+ Long threshold = pulsar.getAdminClient().topicPolicies()
+ .getCompactionThreshold(TOPIC);
+ if (threshold == null || threshold == 0) {
+ pulsar.getAdminClient().topicPolicies()
+ .setCompactionThreshold(TOPIC, COMPACTION_THRESHOLD);
+ log.info("Scheduled compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ } else {
+ log.info("Already set compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ }
+ } catch (PulsarAdminException e) {
+ throw new PulsarServerException("Failed to schedule compaction.", e);
+ }
+ }
+
+ public String getChannelOwner() {
+ CompletableFuture<Optional<LeaderBroker>> future = leaderElectionService.readCurrentLeader();
Review Comment:
I wonder why spotbugs is not complaining about "unconsistent sync" on `leaderElectionService`.
because we still have an access without "synchonized".
I believe that you can add a getter
`private synchronised LeaderElectionService leaderElectionService() { ... }`
and call it here
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,776 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Closed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Constructed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.LeaderElectionServiceStarted;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Started;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ private static final Schema<ServiceUnitStateData> SCHEMA = Schema.JSON(ServiceUnitStateData.class);
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private final String lookupServiceAddress;
+ // TODO: define BrokerRegistry
+ private final Semaphore outstandingCleanupTombstoneMessages;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+ private LeaderElectionService leaderElectionService;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private ScheduledFuture<?> cleanupTasks;
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+ private volatile ChannelState channelState;
+
+ enum ChannelState {
+ Closed(0),
+ Constructed(1),
+ LeaderElectionServiceStarted(2),
+ Started(3);
+
+ ChannelState(int id) {
+ this.id = id;
+ }
+ int id;
+ }
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ ServiceConfiguration conf = pulsar.getConfiguration();
+ this.lookupServiceAddress = pulsar.getLookupServiceAddress();
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder().build();
+ this.cleanupJobs = ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder().build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.channelState = Constructed;
+ }
+
+ public synchronized void start() throws PulsarServerException {
+ try {
Review Comment:
we can add a check on channelState
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,779 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ private static final Schema<ServiceUnitStateData> SCHEMA = Schema.JSON(ServiceUnitStateData.class);
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private String lookupServiceAddress;
+ private LeaderElectionService leaderElectionService;
+ // TODO: define BrokerRegistry
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private Semaphore outstandingPubMessages;
+
+ private Semaphore outstandingCleanupTombstoneMessages;
+ private ScheduledFuture<?> cleanupTasks;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+
+ private ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ ServiceConfiguration conf = pulsar.getConfiguration();
+ this.lookupServiceAddress = pulsar.getAdvertisedAddress() + ":"
+ + (conf.getWebServicePort().isPresent() ? conf.getWebServicePort().get()
+ : conf.getWebServicePortTls().get());
+ this.outstandingPubMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder()
+ .build();
+ this.cleanupJobs =
+ ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder()
+ .build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ }
+
+ public void start() throws PulsarServerException {
+
+ try {
+ if (leaderElectionService != null) {
+ leaderElectionService.close();
+ log.debug("Closed the channel leader election service.");
+ }
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} was elected as the leader."
+ + "Current channel leader is {}",
+ lookupServiceAddress,
+ leaderElectionService.getCurrentLeader());
+ } else {
+ if (leaderElectionService != null) {
+ log.debug("This broker:{} is a follower. "
+ + "Current channel leader is {}",
+ lookupServiceAddress,
+ leaderElectionService.getCurrentLeader());
+ }
+ }
+ });
+ leaderElectionService.start();
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(SCHEMA)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(SCHEMA)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ // TODO: schedule cleanupTasks by brokerRegistry
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+
+ public void close() throws PulsarServerException {
+ try {
+ if (leaderElectionService != null) {
+ leaderElectionService.close();
+ leaderElectionService = null;
+ log.debug("Successfully closed the channel leader election service.");
+ }
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public void scheduleCompaction() throws PulsarServerException {
+ try {
+ Long threshold = pulsar.getAdminClient().topicPolicies()
+ .getCompactionThreshold(TOPIC);
+ if (threshold == null || threshold == 0) {
+ pulsar.getAdminClient().topicPolicies()
+ .setCompactionThreshold(TOPIC, COMPACTION_THRESHOLD);
+ log.info("Scheduled compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ } else {
+ log.info("Already set compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ }
+ } catch (PulsarAdminException e) {
+ throw new PulsarServerException("Failed to schedule compaction.", e);
+ }
+ }
+
+ public String getChannelOwner() {
+ CompletableFuture<Optional<LeaderBroker>> future = leaderElectionService.readCurrentLeader();
+ if (!future.isDone()) {
+ return null;
+ }
+ Optional<LeaderBroker> leader = future.join();
+ if (leader.isEmpty()) {
+ return null;
+ }
+ //expecting http://broker-xyz:port
+ String broker = leader.get().getServiceUrl();
+ broker = broker.substring(broker.lastIndexOf('/') + 1);
Review Comment:
+1
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,776 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Closed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Constructed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.LeaderElectionServiceStarted;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Started;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ private static final Schema<ServiceUnitStateData> SCHEMA = Schema.JSON(ServiceUnitStateData.class);
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private final String lookupServiceAddress;
+ // TODO: define BrokerRegistry
+ private final Semaphore outstandingCleanupTombstoneMessages;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+ private LeaderElectionService leaderElectionService;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private ScheduledFuture<?> cleanupTasks;
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+ private volatile ChannelState channelState;
+
+ enum ChannelState {
+ Closed(0),
+ Constructed(1),
+ LeaderElectionServiceStarted(2),
+ Started(3);
+
+ ChannelState(int id) {
+ this.id = id;
+ }
+ int id;
+ }
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ ServiceConfiguration conf = pulsar.getConfiguration();
+ this.lookupServiceAddress = pulsar.getLookupServiceAddress();
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder().build();
+ this.cleanupJobs = ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder().build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.channelState = Constructed;
+ }
+
+ public synchronized void start() throws PulsarServerException {
+ try {
+ if (leaderElectionService != null) {
+ leaderElectionService.close();
+ log.debug("Closed the channel leader election service.");
+ }
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} is the leader now.", lookupServiceAddress);
+ // TODO: schedule monitorOwnerships by brokerRegistry
+ } else {
+ log.debug("This broker:{} is a follower now.", lookupServiceAddress);
+ // TODO: cancel scheduled monitorOwnerships if any
+ }
+ });
+ leaderElectionService.start();
+ this.channelState = LeaderElectionServiceStarted;
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(SCHEMA)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(SCHEMA)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ channelState = Started;
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public synchronized void close() throws PulsarServerException {
+ channelState = Closed;
+ try {
+ if (leaderElectionService != null) {
+ leaderElectionService.close();
+ leaderElectionService = null;
+ log.debug("Successfully closed the channel leader election service.");
+ }
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ private void validateChannelState(ChannelState targetState) {
+ if (channelState.id < targetState.id) {
+ throw new IllegalStateException("Invalid channel state:" + channelState.name());
+ }
+ }
+
+ public synchronized void scheduleCompaction() throws PulsarServerException {
+ try {
+ Long threshold = pulsar.getAdminClient().topicPolicies()
+ .getCompactionThreshold(TOPIC);
+ if (threshold == null || threshold == 0) {
+ pulsar.getAdminClient().topicPolicies()
+ .setCompactionThreshold(TOPIC, COMPACTION_THRESHOLD);
+ log.info("Scheduled compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ } else {
+ log.info("Already set compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ }
+ } catch (PulsarAdminException e) {
+ throw new PulsarServerException("Failed to schedule compaction.", e);
+ }
+ }
+
+ public String getChannelOwner() {
+ validateChannelState(LeaderElectionServiceStarted);
+ CompletableFuture<Optional<LeaderBroker>> future = leaderElectionService.readCurrentLeader();
+ if (!future.isDone()) {
+ return null;
+ }
+ Optional<LeaderBroker> leader = future.join();
+ if (leader.isEmpty()) {
+ return null;
+ }
+ //expecting http://broker-xyz:port
+ // TODO: discard this protocol prefix removal by a util func that returns lookupServiceAddress(serviceUrl)
+ String broker = leader.get().getServiceUrl();
+ broker = broker.substring(broker.lastIndexOf('/') + 1);
+ return broker;
+ }
+
+ public boolean isChannelOwner() {
+ return isTargetBroker(getChannelOwner());
+ }
+
+ public CompletableFuture<String> getOwnerAsync(String serviceUnit) {
+ validateChannelState(Started);
+ ServiceUnitStateData data = tableview.get(serviceUnit);
+ if (data == null) {
+ return null;
+ }
+ switch (data.state()) {
+ case Owned, Splitting -> {
+ return CompletableFuture.completedFuture(data.broker());
+ }
+ case Assigned, Released -> {
+ return deferGetOwnerRequest(serviceUnit);
+ }
+ default -> {
+ return null;
+ }
+ }
+ }
+
+ public CompletableFuture<String> publishAssignEventAsync(String serviceUnit, String broker) {
+ CompletableFuture<String> getOwnerRequest = deferGetOwnerRequest(serviceUnit);
+ pubAsync(serviceUnit, new ServiceUnitStateData(Assigned, broker))
+ .whenComplete((__, ex) -> {
+ if (ex != null) {
+ getOwnerRequests.remove(serviceUnit);
+ if (!getOwnerRequest.isCompletedExceptionally()) {
+ getOwnerRequest.completeExceptionally(ex);
+ }
+ }
+ });
+
+ return getOwnerRequest;
+ }
+
+ public CompletableFuture<Void> publishUnloadEventAsync(Unload unload) {
+ String serviceUnit = unload.serviceUnit();
+ if (isTransferCommand(unload)) {
+ ServiceUnitStateData next = new ServiceUnitStateData(Assigned,
+ unload.destBroker().get(), unload.sourceBroker());
+ return pubAsync(serviceUnit, next).thenAccept(__ -> {
+ });
+ }
+ return tombstoneAsync(serviceUnit).thenAccept(__ -> {
+ });
+ }
+
+ public CompletableFuture<Void> publishSplitEventAsync(Split split) {
+ String serviceUnit = split.serviceUnit();
+ ServiceUnitStateData next = new ServiceUnitStateData(Splitting, split.sourceBroker());
+ return pubAsync(serviceUnit, next).thenAccept(__ -> {
+ });
+ }
+
+ private void handle(String serviceUnit, ServiceUnitStateData data) {
+ if (log.isDebugEnabled()) {
+ log.info("{} received a handle request for serviceUnit:{}, data:{}",
+ lookupServiceAddress, serviceUnit, data);
+ }
+
+ ServiceUnitState state = data == null ? ServiceUnitState.Free : data.state();
+
+ // TODO : Add state validation in tableview by the compaction strategy
+ switch (state) {
+ case Owned -> handleOwnEvent(serviceUnit, data);
+ case Assigned -> handleAssignEvent(serviceUnit, data);
+ case Released -> handleReleaseEvent(serviceUnit, data);
+ case Splitting -> handleSplitEvent(serviceUnit, data);
+ case Free -> handleFreeEvent(serviceUnit);
+ default -> throw new IllegalStateException("Failed to handle channel data:" + data);
+ }
+ }
+
+ private static boolean isTransferCommand(ServiceUnitStateData data) {
+ if (data == null) {
+ return false;
+ }
+ return StringUtils.isNotEmpty(data.sourceBroker());
+ }
+
+ private static boolean isTransferCommand(Unload data) {
+ return data.destBroker().isPresent();
+ }
+
+ private static String getLogEventTag(ServiceUnitStateData data) {
+ return data == null ? "Free" :
+ isTransferCommand(data) ? "Transfer:" + data.state() : data.state().toString();
+ }
+
+ private void log(Throwable e, String serviceUnit, ServiceUnitStateData data, ServiceUnitStateData next) {
+ if (e == null) {
+ if (log.isDebugEnabled() || isTransferCommand(data)) {
+ log.info("{} handled {} event for serviceUnit:{}, cur:{}, next:{}",
+ lookupServiceAddress, getLogEventTag(data), serviceUnit,
+ data == null ? "" : data,
+ next == null ? "" : next);
+ }
+ } else {
+ log.error("{} failed to handle {} event for serviceUnit:{}, cur:{}, next:{}",
+ lookupServiceAddress, getLogEventTag(data), serviceUnit,
+ data == null ? "" : data,
+ next == null ? "" : next,
+ e);
+ }
+ }
+
+ private void handleOwnEvent(String serviceUnit, ServiceUnitStateData data) {
+ var getOwnerRequest = getOwnerRequests.remove(serviceUnit);
+ if (getOwnerRequest != null) {
+ getOwnerRequest.complete(data.broker());
+ }
+ if (isTargetBroker(data.broker())) {
+ log(null, serviceUnit, data, null);
+ }
+
+ }
+
+ private void handleAssignEvent(String serviceUnit, ServiceUnitStateData data) {
+ deferGetOwnerRequest(serviceUnit);
+ if (isTargetBroker(data.broker())) {
+ ServiceUnitStateData next = new ServiceUnitStateData(
+ isTransferCommand(data) ? Released : Owned, data.broker(), data.sourceBroker());
+ pubAsync(serviceUnit, next)
+ .whenComplete((__, e) -> log(e, serviceUnit, data, next));
+ }
+ }
+
+ private void handleReleaseEvent(String serviceUnit, ServiceUnitStateData data) {
+
+ if (isTargetBroker(data.sourceBroker())) {
+ ServiceUnitStateData next = new ServiceUnitStateData(Owned, data.broker(), data.sourceBroker());
+ // TODO: when close, pass message to clients to connect to the new broker
+ closeServiceUnit(serviceUnit)
+ .thenCompose(__ -> pubAsync(serviceUnit, next))
+ .whenComplete((__, e) -> log(e, serviceUnit, data, next));
+ }
+ }
+
+ private void handleSplitEvent(String serviceUnit, ServiceUnitStateData data) {
+ if (isTargetBroker(data.broker())) {
+ splitServiceUnit(serviceUnit)
+ .thenCompose(__ -> tombstoneAsync(serviceUnit))
+ .whenComplete((__, e) -> log(e, serviceUnit, data, null));
+ }
+ }
+
+ private void handleFreeEvent(String serviceUnit) {
+ closeServiceUnit(serviceUnit)
+ .thenAccept(__ -> {
+ var request = getOwnerRequests.remove(serviceUnit);
+ if (request != null) {
+ request.completeExceptionally(new IllegalStateException("The ownership has been unloaded. "
+ + "No owner is found for serviceUnit: " + serviceUnit));
+ }
+ })
+ .whenComplete((__, e) -> log(e, serviceUnit, null, null));
+ }
+
+ private CompletableFuture<MessageId> pubAsync(String serviceUnit, ServiceUnitStateData data) {
+ validateChannelState(Started);
+ CompletableFuture<MessageId> future = new CompletableFuture<>();
+ producer.newMessage()
+ .key(serviceUnit)
+ .value(data)
+ .sendAsync()
+ .whenComplete((messageId, e) -> {
+ if (e != null) {
+ future.completeExceptionally(e);
Review Comment:
what about logging the error here ?
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,779 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ private static final Schema<ServiceUnitStateData> SCHEMA = Schema.JSON(ServiceUnitStateData.class);
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private String lookupServiceAddress;
+ private LeaderElectionService leaderElectionService;
+ // TODO: define BrokerRegistry
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private Semaphore outstandingPubMessages;
+
+ private Semaphore outstandingCleanupTombstoneMessages;
+ private ScheduledFuture<?> cleanupTasks;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+
+ private ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ ServiceConfiguration conf = pulsar.getConfiguration();
+ this.lookupServiceAddress = pulsar.getAdvertisedAddress() + ":"
+ + (conf.getWebServicePort().isPresent() ? conf.getWebServicePort().get()
+ : conf.getWebServicePortTls().get());
+ this.outstandingPubMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder()
+ .build();
+ this.cleanupJobs =
+ ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder()
+ .build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ }
+
+ public void start() throws PulsarServerException {
+
+ try {
+ if (leaderElectionService != null) {
+ leaderElectionService.close();
+ log.debug("Closed the channel leader election service.");
+ }
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} was elected as the leader."
+ + "Current channel leader is {}",
+ lookupServiceAddress,
+ leaderElectionService.getCurrentLeader());
+ } else {
+ if (leaderElectionService != null) {
+ log.debug("This broker:{} is a follower. "
+ + "Current channel leader is {}",
+ lookupServiceAddress,
+ leaderElectionService.getCurrentLeader());
+ }
+ }
+ });
+ leaderElectionService.start();
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(SCHEMA)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(SCHEMA)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ // TODO: schedule cleanupTasks by brokerRegistry
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+
+ public void close() throws PulsarServerException {
+ try {
+ if (leaderElectionService != null) {
+ leaderElectionService.close();
+ leaderElectionService = null;
+ log.debug("Successfully closed the channel leader election service.");
+ }
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public void scheduleCompaction() throws PulsarServerException {
+ try {
+ Long threshold = pulsar.getAdminClient().topicPolicies()
+ .getCompactionThreshold(TOPIC);
+ if (threshold == null || threshold == 0) {
+ pulsar.getAdminClient().topicPolicies()
+ .setCompactionThreshold(TOPIC, COMPACTION_THRESHOLD);
+ log.info("Scheduled compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ } else {
+ log.info("Already set compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ }
+ } catch (PulsarAdminException e) {
+ throw new PulsarServerException("Failed to schedule compaction.", e);
+ }
+ }
+
+ public String getChannelOwner() {
+ CompletableFuture<Optional<LeaderBroker>> future = leaderElectionService.readCurrentLeader();
Review Comment:
why can't we make it "final" and assign it in the constructor, without calling "start" in the constructor?
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,776 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Closed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Constructed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.LeaderElectionServiceStarted;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Started;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ private static final Schema<ServiceUnitStateData> SCHEMA = Schema.JSON(ServiceUnitStateData.class);
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private final String lookupServiceAddress;
+ // TODO: define BrokerRegistry
+ private final Semaphore outstandingCleanupTombstoneMessages;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+ private LeaderElectionService leaderElectionService;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private ScheduledFuture<?> cleanupTasks;
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+ private volatile ChannelState channelState;
+
+ enum ChannelState {
+ Closed(0),
+ Constructed(1),
+ LeaderElectionServiceStarted(2),
+ Started(3);
+
+ ChannelState(int id) {
+ this.id = id;
+ }
+ int id;
+ }
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ ServiceConfiguration conf = pulsar.getConfiguration();
+ this.lookupServiceAddress = pulsar.getLookupServiceAddress();
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder().build();
+ this.cleanupJobs = ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder().build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.channelState = Constructed;
+ }
+
+ public synchronized void start() throws PulsarServerException {
+ try {
+ if (leaderElectionService != null) {
+ leaderElectionService.close();
+ log.debug("Closed the channel leader election service.");
+ }
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} is the leader now.", lookupServiceAddress);
+ // TODO: schedule monitorOwnerships by brokerRegistry
+ } else {
+ log.debug("This broker:{} is a follower now.", lookupServiceAddress);
+ // TODO: cancel scheduled monitorOwnerships if any
+ }
+ });
+ leaderElectionService.start();
+ this.channelState = LeaderElectionServiceStarted;
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(SCHEMA)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(SCHEMA)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ channelState = Started;
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public synchronized void close() throws PulsarServerException {
+ channelState = Closed;
+ try {
+ if (leaderElectionService != null) {
+ leaderElectionService.close();
+ leaderElectionService = null;
+ log.debug("Successfully closed the channel leader election service.");
+ }
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ private void validateChannelState(ChannelState targetState) {
+ if (channelState.id < targetState.id) {
+ throw new IllegalStateException("Invalid channel state:" + channelState.name());
+ }
+ }
+
+ public synchronized void scheduleCompaction() throws PulsarServerException {
+ try {
+ Long threshold = pulsar.getAdminClient().topicPolicies()
+ .getCompactionThreshold(TOPIC);
+ if (threshold == null || threshold == 0) {
+ pulsar.getAdminClient().topicPolicies()
+ .setCompactionThreshold(TOPIC, COMPACTION_THRESHOLD);
+ log.info("Scheduled compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ } else {
+ log.info("Already set compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ }
+ } catch (PulsarAdminException e) {
+ throw new PulsarServerException("Failed to schedule compaction.", e);
+ }
+ }
+
+ public String getChannelOwner() {
+ validateChannelState(LeaderElectionServiceStarted);
+ CompletableFuture<Optional<LeaderBroker>> future = leaderElectionService.readCurrentLeader();
+ if (!future.isDone()) {
+ return null;
+ }
+ Optional<LeaderBroker> leader = future.join();
+ if (leader.isEmpty()) {
+ return null;
+ }
+ //expecting http://broker-xyz:port
+ // TODO: discard this protocol prefix removal by a util func that returns lookupServiceAddress(serviceUrl)
+ String broker = leader.get().getServiceUrl();
+ broker = broker.substring(broker.lastIndexOf('/') + 1);
+ return broker;
+ }
+
+ public boolean isChannelOwner() {
+ return isTargetBroker(getChannelOwner());
+ }
+
+ public CompletableFuture<String> getOwnerAsync(String serviceUnit) {
+ validateChannelState(Started);
+ ServiceUnitStateData data = tableview.get(serviceUnit);
+ if (data == null) {
+ return null;
+ }
+ switch (data.state()) {
+ case Owned, Splitting -> {
+ return CompletableFuture.completedFuture(data.broker());
+ }
+ case Assigned, Released -> {
+ return deferGetOwnerRequest(serviceUnit);
+ }
+ default -> {
+ return null;
+ }
+ }
+ }
+
+ public CompletableFuture<String> publishAssignEventAsync(String serviceUnit, String broker) {
+ CompletableFuture<String> getOwnerRequest = deferGetOwnerRequest(serviceUnit);
+ pubAsync(serviceUnit, new ServiceUnitStateData(Assigned, broker))
+ .whenComplete((__, ex) -> {
+ if (ex != null) {
+ getOwnerRequests.remove(serviceUnit);
+ if (!getOwnerRequest.isCompletedExceptionally()) {
+ getOwnerRequest.completeExceptionally(ex);
+ }
+ }
+ });
+
+ return getOwnerRequest;
+ }
+
+ public CompletableFuture<Void> publishUnloadEventAsync(Unload unload) {
+ String serviceUnit = unload.serviceUnit();
+ if (isTransferCommand(unload)) {
+ ServiceUnitStateData next = new ServiceUnitStateData(Assigned,
+ unload.destBroker().get(), unload.sourceBroker());
+ return pubAsync(serviceUnit, next).thenAccept(__ -> {
+ });
+ }
+ return tombstoneAsync(serviceUnit).thenAccept(__ -> {
+ });
+ }
+
+ public CompletableFuture<Void> publishSplitEventAsync(Split split) {
+ String serviceUnit = split.serviceUnit();
+ ServiceUnitStateData next = new ServiceUnitStateData(Splitting, split.sourceBroker());
+ return pubAsync(serviceUnit, next).thenAccept(__ -> {
+ });
+ }
+
+ private void handle(String serviceUnit, ServiceUnitStateData data) {
+ if (log.isDebugEnabled()) {
+ log.info("{} received a handle request for serviceUnit:{}, data:{}",
+ lookupServiceAddress, serviceUnit, data);
+ }
+
+ ServiceUnitState state = data == null ? ServiceUnitState.Free : data.state();
+
+ // TODO : Add state validation in tableview by the compaction strategy
+ switch (state) {
+ case Owned -> handleOwnEvent(serviceUnit, data);
+ case Assigned -> handleAssignEvent(serviceUnit, data);
+ case Released -> handleReleaseEvent(serviceUnit, data);
+ case Splitting -> handleSplitEvent(serviceUnit, data);
+ case Free -> handleFreeEvent(serviceUnit);
+ default -> throw new IllegalStateException("Failed to handle channel data:" + data);
+ }
+ }
+
+ private static boolean isTransferCommand(ServiceUnitStateData data) {
+ if (data == null) {
+ return false;
+ }
+ return StringUtils.isNotEmpty(data.sourceBroker());
+ }
+
+ private static boolean isTransferCommand(Unload data) {
+ return data.destBroker().isPresent();
+ }
+
+ private static String getLogEventTag(ServiceUnitStateData data) {
+ return data == null ? "Free" :
+ isTransferCommand(data) ? "Transfer:" + data.state() : data.state().toString();
+ }
+
+ private void log(Throwable e, String serviceUnit, ServiceUnitStateData data, ServiceUnitStateData next) {
+ if (e == null) {
+ if (log.isDebugEnabled() || isTransferCommand(data)) {
+ log.info("{} handled {} event for serviceUnit:{}, cur:{}, next:{}",
+ lookupServiceAddress, getLogEventTag(data), serviceUnit,
+ data == null ? "" : data,
+ next == null ? "" : next);
+ }
+ } else {
+ log.error("{} failed to handle {} event for serviceUnit:{}, cur:{}, next:{}",
+ lookupServiceAddress, getLogEventTag(data), serviceUnit,
+ data == null ? "" : data,
+ next == null ? "" : next,
+ e);
+ }
+ }
+
+ private void handleOwnEvent(String serviceUnit, ServiceUnitStateData data) {
+ var getOwnerRequest = getOwnerRequests.remove(serviceUnit);
+ if (getOwnerRequest != null) {
+ getOwnerRequest.complete(data.broker());
+ }
+ if (isTargetBroker(data.broker())) {
+ log(null, serviceUnit, data, null);
+ }
+
+ }
+
+ private void handleAssignEvent(String serviceUnit, ServiceUnitStateData data) {
+ deferGetOwnerRequest(serviceUnit);
+ if (isTargetBroker(data.broker())) {
+ ServiceUnitStateData next = new ServiceUnitStateData(
+ isTransferCommand(data) ? Released : Owned, data.broker(), data.sourceBroker());
+ pubAsync(serviceUnit, next)
+ .whenComplete((__, e) -> log(e, serviceUnit, data, next));
+ }
+ }
+
+ private void handleReleaseEvent(String serviceUnit, ServiceUnitStateData data) {
+
+ if (isTargetBroker(data.sourceBroker())) {
+ ServiceUnitStateData next = new ServiceUnitStateData(Owned, data.broker(), data.sourceBroker());
+ // TODO: when close, pass message to clients to connect to the new broker
+ closeServiceUnit(serviceUnit)
+ .thenCompose(__ -> pubAsync(serviceUnit, next))
+ .whenComplete((__, e) -> log(e, serviceUnit, data, next));
+ }
+ }
+
+ private void handleSplitEvent(String serviceUnit, ServiceUnitStateData data) {
+ if (isTargetBroker(data.broker())) {
+ splitServiceUnit(serviceUnit)
+ .thenCompose(__ -> tombstoneAsync(serviceUnit))
+ .whenComplete((__, e) -> log(e, serviceUnit, data, null));
+ }
+ }
+
+ private void handleFreeEvent(String serviceUnit) {
+ closeServiceUnit(serviceUnit)
+ .thenAccept(__ -> {
+ var request = getOwnerRequests.remove(serviceUnit);
+ if (request != null) {
+ request.completeExceptionally(new IllegalStateException("The ownership has been unloaded. "
+ + "No owner is found for serviceUnit: " + serviceUnit));
+ }
+ })
+ .whenComplete((__, e) -> log(e, serviceUnit, null, null));
+ }
+
+ private CompletableFuture<MessageId> pubAsync(String serviceUnit, ServiceUnitStateData data) {
+ validateChannelState(Started);
+ CompletableFuture<MessageId> future = new CompletableFuture<>();
+ producer.newMessage()
+ .key(serviceUnit)
+ .value(data)
+ .sendAsync()
+ .whenComplete((messageId, e) -> {
+ if (e != null) {
+ future.completeExceptionally(e);
+ } else {
+ future.complete(messageId);
+ }
+ });
+ return future;
+ }
+
+ private CompletableFuture<MessageId> tombstoneAsync(String serviceUnit) {
+ return pubAsync(serviceUnit, null);
+ }
+
+ private boolean isTargetBroker(String broker) {
+ if (broker == null) {
+ return false;
+ }
+ return broker.equals(lookupServiceAddress);
+ }
+
+ private NamespaceBundle getNamespaceBundle(String bundle) {
+ final String namespaceName = LoadManagerShared.getNamespaceNameFromBundleName(bundle);
+ final String bundleRange = LoadManagerShared.getBundleRangeFromBundleName(bundle);
+ return pulsar.getNamespaceService().getNamespaceBundleFactory().getBundle(namespaceName, bundleRange);
+ }
+
+ private CompletableFuture<String> deferGetOwnerRequest(String serviceUnit) {
+ return getOwnerRequests
+ .computeIfAbsent(serviceUnit, k -> {
+ CompletableFuture future = new CompletableFuture<>();
+ future.orTimeout(inFlightStateWaitingTimeInMillis, TimeUnit.MILLISECONDS)
+ .whenComplete((v, e) -> {
+ if (e != null) {
+ getOwnerRequests.remove(serviceUnit);
+ log.warn("Failed to getOwner for serviceUnit:{}",
+ serviceUnit, e);
+ future.completeExceptionally((Throwable) e);
+ }
+ }
+ );
+ return future;
+ });
+ }
+
+ private CompletableFuture<Integer> closeServiceUnit(String serviceUnit) {
+ long startTime = System.nanoTime();
+ MutableInt unloadedTopics = new MutableInt();
+ NamespaceBundle bundle = getNamespaceBundle(serviceUnit);
+ return pulsar.getBrokerService().unloadServiceUnit(
+ bundle,
+ false,
+ pulsar.getConfig().getNamespaceBundleUnloadingTimeoutMs(),
+ TimeUnit.MILLISECONDS)
+ .handle((numUnloadedTopics, ex) -> {
+ if (ex != null) {
+ // ignore topic-close failure to unload bundle
+ log.error("Failed to close topics under bundle:{}",
+ bundle.toString(), ex);
+ } else {
+ unloadedTopics.setValue(numUnloadedTopics);
+ }
+ // clean up topics that failed to unload from the broker ownership cache
+ pulsar.getBrokerService().cleanUnloadedTopicFromCache(bundle);
+ return numUnloadedTopics;
+ })
+ .whenComplete((ignored, ex) -> {
+ double unloadBundleTime = TimeUnit.NANOSECONDS
+ .toMillis((System.nanoTime() - startTime));
+ log.info("Unloading bundle:{} with {} topics completed in {} ms",
+ bundle, unloadedTopics, unloadBundleTime, ex);
+ });
+ }
+
+ private CompletableFuture<Void> splitServiceUnit(String serviceUnit) {
+ // TODO: after the split we need to write the child ownerships to BSC instead of ZK.
+ // Use the configured NamespaceBundleSplitAlgorithm
+ long startTime = System.nanoTime();
+ return pulsar.getNamespaceService()
+ .splitAndOwnBundle(getNamespaceBundle(serviceUnit),
+ false,
+ NamespaceBundleSplitAlgorithm.RANGE_EQUALLY_DIVIDE_ALGO,
+ null)
+ .whenComplete((__, ex) -> {
+ double splitBundleTime = TimeUnit.NANOSECONDS
+ .toMillis((System.nanoTime() - startTime));
+ log.info("Splitting {} namespace-bundle completed in {} ms",
Review Comment:
we should log it as ERROR and not "info" in case ex is not null (and log the error)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] heesung-sn commented on a diff in pull request #18489: [improve][broker] PIP-192 Added ServiceUnitStateChannelImpl
Posted by GitBox <gi...@apache.org>.
heesung-sn commented on code in PR #18489:
URL: https://github.com/apache/pulsar/pull/18489#discussion_r1034310909
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,778 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Free;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Closed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Constructed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.LeaderElectionServiceStarted;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Started;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private final Schema<ServiceUnitStateData> schema;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private final String lookupServiceAddress;
+ // TODO: define BrokerRegistry
+ private final Semaphore outstandingCleanupTombstoneMessages;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+ private final LeaderElectionService leaderElectionService;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private ScheduledFuture<?> cleanupTasks;
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+ private volatile ChannelState channelState;
+
+ enum ChannelState {
+ Closed(0),
+ Constructed(1),
+ LeaderElectionServiceStarted(2),
+ Started(3);
+
+ ChannelState(int id) {
+ this.id = id;
+ }
+ int id;
+ }
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ this.lookupServiceAddress = pulsar.getLookupServiceAddress();
+ this.schema = Schema.JSON(ServiceUnitStateData.class);
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder().build();
+ this.cleanupJobs = ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder().build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} is the leader now.", lookupServiceAddress);
+ // TODO: schedule monitorOwnerships by brokerRegistry
+ } else {
+ log.debug("This broker:{} is a follower now.", lookupServiceAddress);
+ // TODO: cancel scheduled monitorOwnerships if any
+ }
+ });
+ this.channelState = Constructed;
+ }
+
+ public synchronized void start() throws PulsarServerException {
+ validateChannelState(LeaderElectionServiceStarted, false);
+ try {
+ leaderElectionService.start();
+ this.channelState = LeaderElectionServiceStarted;
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(schema)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(schema)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ channelState = Started;
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public synchronized void close() throws PulsarServerException {
+ channelState = Closed;
+ try {
+ leaderElectionService.close();
+ log.debug("Successfully closed the channel leader election service.");
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ private void validateChannelState(ChannelState targetState, boolean checkLowerIds) {
+ int order = checkLowerIds ? -1 : 1;
+ if (Integer.compare(channelState.id, targetState.id) * order > 0) {
+ throw new IllegalStateException("Invalid channel state:" + channelState.name());
+ }
+ }
+
+ public synchronized void scheduleCompaction() throws PulsarServerException {
Review Comment:
From my understanding, this needs to be called separately when the topic policy is ready.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] heesung-sn commented on a diff in pull request #18489: [improve][broker] PIP-192 Added ServiceUnitStateChannelImpl
Posted by GitBox <gi...@apache.org>.
heesung-sn commented on code in PR #18489:
URL: https://github.com/apache/pulsar/pull/18489#discussion_r1034325806
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,778 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Free;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Closed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Constructed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.LeaderElectionServiceStarted;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Started;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private final Schema<ServiceUnitStateData> schema;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private final String lookupServiceAddress;
+ // TODO: define BrokerRegistry
+ private final Semaphore outstandingCleanupTombstoneMessages;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+ private final LeaderElectionService leaderElectionService;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private ScheduledFuture<?> cleanupTasks;
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+ private volatile ChannelState channelState;
+
+ enum ChannelState {
+ Closed(0),
+ Constructed(1),
+ LeaderElectionServiceStarted(2),
+ Started(3);
+
+ ChannelState(int id) {
+ this.id = id;
+ }
+ int id;
+ }
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ this.lookupServiceAddress = pulsar.getLookupServiceAddress();
+ this.schema = Schema.JSON(ServiceUnitStateData.class);
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder().build();
+ this.cleanupJobs = ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder().build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} is the leader now.", lookupServiceAddress);
+ // TODO: schedule monitorOwnerships by brokerRegistry
+ } else {
+ log.debug("This broker:{} is a follower now.", lookupServiceAddress);
+ // TODO: cancel scheduled monitorOwnerships if any
+ }
+ });
+ this.channelState = Constructed;
+ }
+
+ public synchronized void start() throws PulsarServerException {
+ validateChannelState(LeaderElectionServiceStarted, false);
+ try {
+ leaderElectionService.start();
+ this.channelState = LeaderElectionServiceStarted;
+ log.debug("Successfully started the channel leader election service.");
+
+ if (producer != null) {
+ producer.close();
+ log.debug("Closed the channel producer.");
+ }
+ producer = pulsar.getClient().newProducer(schema)
+ .enableBatching(true)
+ .topic(TOPIC)
+ .create();
+
+ log.debug("Successfully started the channel producer.");
+
+ if (tableview != null) {
+ tableview.close();
+ log.debug("Closed the channel tableview.");
+ }
+ tableview = pulsar.getClient().newTableViewBuilder(schema)
+ .topic(TOPIC)
+ // TODO: enable CompactionStrategy
+ .create();
+ // TODO: schedule listen instead of foreachAndListen
+ tableview.forEachAndListen((key, value) -> handle(key, value));
+ log.debug("Successfully started the channel tableview.");
+
+ pulsar.getLocalMetadataStore().registerSessionListener(this::handleMetadataSessionEvent);
+ log.debug("Successfully registered the handleMetadataSessionEvent");
+
+ channelState = Started;
+ log.info("Successfully started the channel.");
+ } catch (Exception e) {
+ String msg = "Failed to start the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ public synchronized void close() throws PulsarServerException {
+ channelState = Closed;
+ try {
+ leaderElectionService.close();
+ log.debug("Successfully closed the channel leader election service.");
+
+ if (tableview != null) {
+ tableview.close();
+ tableview = null;
+ log.debug("Successfully closed the channel tableview.");
+ }
+
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ log.info("Successfully closed the channel producer.");
+ }
+
+ // TODO: clean brokerRegistry
+
+ if (cleanupTasks != null) {
+ cleanupTasks.cancel(true);
+ cleanupTasks = null;
+ log.info("Successfully cancelled the cleanup tasks");
+ }
+
+ log.info("Successfully closed the channel.");
+
+ } catch (Exception e) {
+ String msg = "Failed to close the channel.";
+ log.error(msg, e);
+ throw new PulsarServerException(msg, e);
+ }
+ }
+
+ private void validateChannelState(ChannelState targetState, boolean checkLowerIds) {
+ int order = checkLowerIds ? -1 : 1;
+ if (Integer.compare(channelState.id, targetState.id) * order > 0) {
+ throw new IllegalStateException("Invalid channel state:" + channelState.name());
+ }
+ }
+
+ public synchronized void scheduleCompaction() throws PulsarServerException {
+ try {
+ Long threshold = pulsar.getAdminClient().topicPolicies()
+ .getCompactionThreshold(TOPIC);
+ if (threshold == null || threshold == 0) {
+ pulsar.getAdminClient().topicPolicies()
+ .setCompactionThreshold(TOPIC, COMPACTION_THRESHOLD);
+ log.info("Scheduled compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ } else {
+ log.info("Already set compaction on topic:{}, threshold:{} bytes", TOPIC, COMPACTION_THRESHOLD);
+ }
+ } catch (PulsarAdminException e) {
+ throw new PulsarServerException("Failed to schedule compaction.", e);
+ }
+ }
+
+ public String getChannelOwner() {
+ validateChannelState(LeaderElectionServiceStarted, true);
+ CompletableFuture<Optional<LeaderBroker>> future = leaderElectionService.readCurrentLeader();
+ if (!future.isDone()) {
+ return null;
+ }
+ Optional<LeaderBroker> leader = future.join();
+ if (leader.isEmpty()) {
+ return null;
+ }
+ //expecting http://broker-xyz:port
+ // TODO: discard this protocol prefix removal by a util func that returns lookupServiceAddress(serviceUrl)
+ String broker = leader.get().getServiceUrl();
+ broker = broker.substring(broker.lastIndexOf('/') + 1);
+ return broker;
+ }
+
+ public boolean isChannelOwner() {
+ return isTargetBroker(getChannelOwner());
+ }
+
+ public CompletableFuture<String> getOwnerAsync(String serviceUnit) {
+ validateChannelState(Started, true);
+ ServiceUnitStateData data = tableview.get(serviceUnit);
+ if (data == null) {
+ return null;
+ }
+ switch (data.state()) {
+ case Owned, Splitting -> {
+ return CompletableFuture.completedFuture(data.broker());
+ }
+ case Assigned, Released -> {
+ return deferGetOwnerRequest(serviceUnit);
+ }
+ default -> {
+ return null;
+ }
+ }
+ }
+
+ public CompletableFuture<String> publishAssignEventAsync(String serviceUnit, String broker) {
+ CompletableFuture<String> getOwnerRequest = deferGetOwnerRequest(serviceUnit);
+ pubAsync(serviceUnit, new ServiceUnitStateData(Assigned, broker))
+ .whenComplete((__, ex) -> {
+ if (ex != null) {
+ getOwnerRequests.remove(serviceUnit);
+ if (!getOwnerRequest.isCompletedExceptionally()) {
+ getOwnerRequest.completeExceptionally(ex);
+ }
+ }
+ });
+
+ return getOwnerRequest;
+ }
+
+ public CompletableFuture<Void> publishUnloadEventAsync(Unload unload) {
+ String serviceUnit = unload.serviceUnit();
+ if (isTransferCommand(unload)) {
+ ServiceUnitStateData next = new ServiceUnitStateData(Assigned,
+ unload.destBroker().get(), unload.sourceBroker());
+ return pubAsync(serviceUnit, next).thenAccept(__ -> {
+ });
+ }
+ return tombstoneAsync(serviceUnit).thenAccept(__ -> {
+ });
+ }
+
+ public CompletableFuture<Void> publishSplitEventAsync(Split split) {
+ String serviceUnit = split.serviceUnit();
+ ServiceUnitStateData next = new ServiceUnitStateData(Splitting, split.sourceBroker());
+ return pubAsync(serviceUnit, next).thenAccept(__ -> {
+ });
+ }
+
+ private void handle(String serviceUnit, ServiceUnitStateData data) {
+ if (log.isDebugEnabled()) {
+ log.info("{} received a handle request for serviceUnit:{}, data:{}",
+ lookupServiceAddress, serviceUnit, data);
+ }
+
+ ServiceUnitState state = data == null ? Free : data.state();
+
+ // TODO : Add state validation in tableview by the compaction strategy
+ switch (state) {
+ case Owned -> handleOwnEvent(serviceUnit, data);
+ case Assigned -> handleAssignEvent(serviceUnit, data);
+ case Released -> handleReleaseEvent(serviceUnit, data);
+ case Splitting -> handleSplitEvent(serviceUnit, data);
+ case Free -> handleFreeEvent(serviceUnit);
+ default -> throw new IllegalStateException("Failed to handle channel data:" + data);
+ }
+ }
+
+ private static boolean isTransferCommand(ServiceUnitStateData data) {
+ if (data == null) {
+ return false;
+ }
+ return StringUtils.isNotEmpty(data.sourceBroker());
+ }
+
+ private static boolean isTransferCommand(Unload data) {
+ return data.destBroker().isPresent();
+ }
+
+ private static String getLogEventTag(ServiceUnitStateData data) {
+ return data == null ? "Free" :
+ isTransferCommand(data) ? "Transfer:" + data.state() : data.state().toString();
+ }
+
+ private void log(Throwable e, String serviceUnit, ServiceUnitStateData data, ServiceUnitStateData next) {
+ if (e == null) {
+ if (log.isDebugEnabled() || isTransferCommand(data)) {
+ log.info("{} handled {} event for serviceUnit:{}, cur:{}, next:{}",
+ lookupServiceAddress, getLogEventTag(data), serviceUnit,
+ data == null ? "" : data,
+ next == null ? "" : next);
+ }
+ } else {
+ log.error("{} failed to handle {} event for serviceUnit:{}, cur:{}, next:{}",
+ lookupServiceAddress, getLogEventTag(data), serviceUnit,
+ data == null ? "" : data,
+ next == null ? "" : next,
+ e);
+ }
+ }
+
+ private void handleOwnEvent(String serviceUnit, ServiceUnitStateData data) {
+ var getOwnerRequest = getOwnerRequests.remove(serviceUnit);
+ if (getOwnerRequest != null) {
+ getOwnerRequest.complete(data.broker());
+ }
+ if (isTargetBroker(data.broker())) {
+ log(null, serviceUnit, data, null);
+ }
+
+ }
+
+ private void handleAssignEvent(String serviceUnit, ServiceUnitStateData data) {
+ deferGetOwnerRequest(serviceUnit);
+ if (isTargetBroker(data.broker())) {
+ ServiceUnitStateData next = new ServiceUnitStateData(
+ isTransferCommand(data) ? Released : Owned, data.broker(), data.sourceBroker());
+ pubAsync(serviceUnit, next)
+ .whenComplete((__, e) -> log(e, serviceUnit, data, next));
+ }
+ }
+
+ private void handleReleaseEvent(String serviceUnit, ServiceUnitStateData data) {
+
+ if (isTargetBroker(data.sourceBroker())) {
+ ServiceUnitStateData next = new ServiceUnitStateData(Owned, data.broker(), data.sourceBroker());
+ // TODO: when close, pass message to clients to connect to the new broker
+ closeServiceUnit(serviceUnit)
+ .thenCompose(__ -> pubAsync(serviceUnit, next))
+ .whenComplete((__, e) -> log(e, serviceUnit, data, next));
+ }
+ }
+
+ private void handleSplitEvent(String serviceUnit, ServiceUnitStateData data) {
+ if (isTargetBroker(data.broker())) {
+ splitServiceUnit(serviceUnit)
+ .thenCompose(__ -> tombstoneAsync(serviceUnit))
+ .whenComplete((__, e) -> log(e, serviceUnit, data, null));
+ }
+ }
+
+ private void handleFreeEvent(String serviceUnit) {
+ closeServiceUnit(serviceUnit)
+ .thenAccept(__ -> {
+ var request = getOwnerRequests.remove(serviceUnit);
+ if (request != null) {
+ request.completeExceptionally(new IllegalStateException("The ownership has been unloaded. "
+ + "No owner is found for serviceUnit: " + serviceUnit));
+ }
+ })
+ .whenComplete((__, e) -> log(e, serviceUnit, null, null));
+ }
+
+ private CompletableFuture<MessageId> pubAsync(String serviceUnit, ServiceUnitStateData data) {
+ validateChannelState(Started, true);
+ CompletableFuture<MessageId> future = new CompletableFuture<>();
+ producer.newMessage()
+ .key(serviceUnit)
+ .value(data)
+ .sendAsync()
+ .whenComplete((messageId, e) -> {
+ if (e != null) {
+ log.error("Failed to publish the message: serviceUnit:{}, data:{}",
+ serviceUnit, data, e);
+ future.completeExceptionally(e);
+ } else {
+ future.complete(messageId);
+ }
+ });
+ return future;
+ }
+
+ private CompletableFuture<MessageId> tombstoneAsync(String serviceUnit) {
+ return pubAsync(serviceUnit, null);
+ }
+
+ private boolean isTargetBroker(String broker) {
+ if (broker == null) {
+ return false;
+ }
+ return broker.equals(lookupServiceAddress);
+ }
+
+ private NamespaceBundle getNamespaceBundle(String bundle) {
+ final String namespaceName = LoadManagerShared.getNamespaceNameFromBundleName(bundle);
+ final String bundleRange = LoadManagerShared.getBundleRangeFromBundleName(bundle);
+ return pulsar.getNamespaceService().getNamespaceBundleFactory().getBundle(namespaceName, bundleRange);
+ }
+
+ private CompletableFuture<String> deferGetOwnerRequest(String serviceUnit) {
+ return getOwnerRequests
+ .computeIfAbsent(serviceUnit, k -> {
+ CompletableFuture future = new CompletableFuture<>();
+ future.orTimeout(inFlightStateWaitingTimeInMillis, TimeUnit.MILLISECONDS)
+ .whenComplete((v, e) -> {
+ if (e != null) {
+ getOwnerRequests.remove(serviceUnit);
+ log.warn("Failed to getOwner for serviceUnit:{}",
+ serviceUnit, e);
+ future.completeExceptionally((Throwable) e);
+ }
+ }
+ );
+ return future;
+ });
+ }
+
+ private CompletableFuture<Integer> closeServiceUnit(String serviceUnit) {
+ long startTime = System.nanoTime();
+ MutableInt unloadedTopics = new MutableInt();
+ NamespaceBundle bundle = getNamespaceBundle(serviceUnit);
+ return pulsar.getBrokerService().unloadServiceUnit(
+ bundle,
+ false,
+ pulsar.getConfig().getNamespaceBundleUnloadingTimeoutMs(),
+ TimeUnit.MILLISECONDS)
+ .handle((numUnloadedTopics, ex) -> {
+ if (ex != null) {
+ // ignore topic-close failure to unload bundle
+ log.error("Failed to close topics under bundle:{}",
+ bundle.toString(), ex);
+ } else {
+ unloadedTopics.setValue(numUnloadedTopics);
+ }
+ // clean up topics that failed to unload from the broker ownership cache
+ pulsar.getBrokerService().cleanUnloadedTopicFromCache(bundle);
+ return numUnloadedTopics;
+ })
+ .whenComplete((ignored, ex) -> {
+ double unloadBundleTime = TimeUnit.NANOSECONDS
+ .toMillis((System.nanoTime() - startTime));
+ log.info("Unloading bundle:{} with {} topics completed in {} ms",
+ bundle, unloadedTopics, unloadBundleTime, ex);
+ });
+ }
+
+ private CompletableFuture<Void> splitServiceUnit(String serviceUnit) {
+ // TODO: after the split we need to write the child ownerships to BSC instead of ZK.
+ // Use the configured NamespaceBundleSplitAlgorithm
+ long startTime = System.nanoTime();
+ return pulsar.getNamespaceService()
+ .splitAndOwnBundle(getNamespaceBundle(serviceUnit),
+ false,
+ NamespaceBundleSplitAlgorithm.RANGE_EQUALLY_DIVIDE_ALGO,
Review Comment:
As described in todo comment, this split logic requires more updates.( Kai will raise a pr for this update). But I can make it use the algo from the user config in this pr.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] heesung-sn commented on a diff in pull request #18489: [improve][broker] PIP-192 Added ServiceUnitStateChannelImpl
Posted by GitBox <gi...@apache.org>.
heesung-sn commented on code in PR #18489:
URL: https://github.com/apache/pulsar/pull/18489#discussion_r1034310124
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -0,0 +1,778 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.channel;
+
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Assigned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Free;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Owned;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Released;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitState.Splitting;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Closed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Constructed;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.LeaderElectionServiceStarted;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.ChannelState.Started;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Jittery;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Stable;
+import static org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl.MetadataState.Unstable;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionLost;
+import static org.apache.pulsar.metadata.api.extended.SessionEvent.SessionReestablished;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.loadbalance.LeaderBroker;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Split;
+import org.apache.pulsar.broker.loadbalance.extensions.models.Unload;
+import org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.NamespaceBundleSplitAlgorithm;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LeaderElectionState;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+@Slf4j
+public class ServiceUnitStateChannelImpl implements ServiceUnitStateChannel {
+ public static final String TOPIC =
+ TopicDomain.persistent.value()
+ + "://"
+ + NamespaceName.SYSTEM_NAMESPACE
+ + "/service-unit-state-channel";
+ // TODO: define StateCompactionStrategy
+ private static final long COMPACTION_THRESHOLD = 5 * 1024 * 1024; // 5mb
+ private static final long MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS = 30 * 1000; // 30sec
+ public static final long MAX_CLEAN_UP_DELAY_TIME_IN_SECS = 3 * 60; // 3 mins
+ private static final long MIN_CLEAN_UP_DELAY_TIME_IN_SECS = 0; // 0 secs to clean immediately
+ private static final int MAX_OUTSTANDING_PUB_MESSAGES = 500;
+ private final PulsarService pulsar;
+ private final Schema<ServiceUnitStateData> schema;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<String>> getOwnerRequests;
+ private final String lookupServiceAddress;
+ // TODO: define BrokerRegistry
+ private final Semaphore outstandingCleanupTombstoneMessages;
+ private final ConcurrentOpenHashMap<String, CompletableFuture<Void>> cleanupJobs;
+ private final LeaderElectionService leaderElectionService;
+ private TableView<ServiceUnitStateData> tableview;
+ private Producer<ServiceUnitStateData> producer;
+ private ScheduledFuture<?> cleanupTasks;
+ private SessionEvent lastMetadataSessionEvent = SessionReestablished;
+ private long lastMetadataSessionEventTimestamp = 0;
+ private long inFlightStateWaitingTimeInMillis;
+ private long maxCleanupDelayTimeInSecs;
+ private long minCleanupDelayTimeInSecs;
+ // cleanup metrics
+ private long totalCleanupCnt = 0;
+ private long totalBrokerCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupTombstoneCnt = 0;
+ private long totalServiceUnitCleanupErrorCnt = 0;
+ private long totalCleanupScheduledCnt = 0;
+ private long totalCleanupIgnoredCnt = 0;
+ private long totalCleanupCancelledCnt = 0;
+ private volatile ChannelState channelState;
+
+ enum ChannelState {
+ Closed(0),
+ Constructed(1),
+ LeaderElectionServiceStarted(2),
+ Started(3);
+
+ ChannelState(int id) {
+ this.id = id;
+ }
+ int id;
+ }
+
+ enum MetadataState {
+ Stable,
+ Jittery,
+ Unstable
+ }
+
+ public ServiceUnitStateChannelImpl(PulsarService pulsar) {
+ this.pulsar = pulsar;
+ this.lookupServiceAddress = pulsar.getLookupServiceAddress();
+ this.schema = Schema.JSON(ServiceUnitStateData.class);
+ this.outstandingCleanupTombstoneMessages = new Semaphore(MAX_OUTSTANDING_PUB_MESSAGES);
+ this.getOwnerRequests = ConcurrentOpenHashMap.<String,
+ CompletableFuture<String>>newBuilder().build();
+ this.cleanupJobs = ConcurrentOpenHashMap.<String, CompletableFuture<Void>>newBuilder().build();
+ this.inFlightStateWaitingTimeInMillis = MAX_IN_FLIGHT_STATE_WAITING_TIME_IN_MILLIS;
+ this.maxCleanupDelayTimeInSecs = MAX_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.minCleanupDelayTimeInSecs = MIN_CLEAN_UP_DELAY_TIME_IN_SECS;
+ this.leaderElectionService = new LeaderElectionService(
+ pulsar.getCoordinationService(), pulsar.getSafeWebServiceAddress(),
+ state -> {
+ if (state == LeaderElectionState.Leading) {
+ log.debug("This broker:{} is the leader now.", lookupServiceAddress);
+ // TODO: schedule monitorOwnerships by brokerRegistry
+ } else {
+ log.debug("This broker:{} is a follower now.", lookupServiceAddress);
+ // TODO: cancel scheduled monitorOwnerships if any
+ }
+ });
+ this.channelState = Constructed;
+ }
+
+ public synchronized void start() throws PulsarServerException {
Review Comment:
Yes, but the channelOwner can own the system topic.(The leaderElectionServiceStarted channel state can return getChannelOwner api.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org