You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "lhotari (via GitHub)" <gi...@apache.org> on 2025/10/09 11:53:11 UTC
[PR] [feat][broker/proxy] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
lhotari opened a new pull request, #24833:
URL: https://github.com/apache/pulsar/pull/24833
[PIP-442](https://github.com/apache/pulsar/blob/master/pip/pip-442.md)
### Motivation
See [PIP-442](https://github.com/apache/pulsar/blob/master/pip/pip-442.md).
### Modifications
- This is the first part of the implementation for PIP-442
- It covers most parts described in PIP-442 document excluding the metrics
- Test for Pulsar Proxy is still missing. This PR is for initial review.
### Documentation
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
- [ ] `doc` <!-- Your PR contains doc changes. -->
- [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
- [x] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2494351811
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/TopicListService.java:
##########
@@ -101,6 +112,7 @@ public void accept(String topicName, NotificationType notificationType) {
private final boolean enableSubscriptionPatternEvaluation;
private final int maxSubscriptionPatternLength;
private final ConcurrentLongHashMap<CompletableFuture<TopicListWatcher>> watchers;
+ private final Backoff retryBackoff;
public TopicListService(PulsarService pulsar, ServerCnx connection,
Review Comment:
> That situation is what I want to say. It will cause consumers to be unable to receive messages that were sent to the new partitions until the client application restarts.
That doesn't hold since the solution will retry with back off in case of errors.
There's some additional changes that I'll need to make.
* There is a need to stop retries
* if the watcher is closed
* the connection is closed
* operations should be handled in order
* retrying could break the ordering currently
I'll follow up with changes to address the above issues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari closed pull request #24833: [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList
URL: https://github.com/apache/pulsar/pull/24833
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2436985881
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/ServerCnx.java:
##########
@@ -2610,6 +2581,99 @@ protected void handleGetTopicsOfNamespace(CommandGetTopicsOfNamespace commandGet
}
}
+ private void internalHandleGetTopicsOfNamespace(String namespace, NamespaceName namespaceName, long requestId,
+ CommandGetTopicsOfNamespace.Mode mode,
+ Optional<String> topicsPattern, Optional<String> topicsHash,
+ Semaphore lookupSemaphore) {
+ BooleanSupplier isPermitRequestCancelled = () -> !ctx().channel().isActive();
+ TopicListSizeResultCache.ResultHolder
+ listSizeHolder = service.getTopicListSizeResultCache().getTopicListSize(namespaceName.toString(), mode);
+ listSizeHolder.getSizeAsync().thenAccept(initialSize -> {
+ maxTopicListInFlightLimiter.withAcquiredPermits(initialSize,
+ AsyncDualMemoryLimiter.LimitType.HEAP_MEMORY, isPermitRequestCancelled, initialPermits -> {
+ return getBrokerService().pulsar().getNamespaceService()
+ .getListOfUserTopics(namespaceName, mode)
+ .thenAccept(topics -> {
+ long actualSize = topics.stream()
+ .mapToInt(ByteBufUtil::utf8Bytes) // convert character count to bytes
+ .map(n -> n + 32) // add 32 bytes overhead for each entry
Review Comment:
In JVMs, an object header typically consumes 16 bytes and a reference consumes 4 bytes or 8 bytes. This is improve the accuracy of the estimation of consumed heap. I guess there's also padding involved.
I think that there's at least 20 bytes of overhead per entry.
One detail here is that the memory limiter doesn't do an accurate calculation about the actual consumed heap. The purpose is to have a way to ensure that there is a limit for heap memory. In many cases the referenced objects are shared and therefore they won't duplicate the usage of heap memory. It would be possible to address that, but it would unnecessarily complicate the solution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2466067526
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/ServerCnx.java:
##########
@@ -2610,6 +2581,93 @@ protected void handleGetTopicsOfNamespace(CommandGetTopicsOfNamespace commandGet
}
}
+ private void internalHandleGetTopicsOfNamespace(String namespace, NamespaceName namespaceName, long requestId,
+ CommandGetTopicsOfNamespace.Mode mode,
+ Optional<String> topicsPattern, Optional<String> topicsHash,
+ Semaphore lookupSemaphore) {
+ BooleanSupplier isPermitRequestCancelled = () -> !ctx().channel().isActive();
+ TopicListSizeResultCache.ResultHolder
+ listSizeHolder = service.getTopicListSizeResultCache().getTopicListSize(namespaceName.toString(), mode);
+ listSizeHolder.getSizeAsync().thenAccept(initialSize -> {
+ maxTopicListInFlightLimiter.withAcquiredPermits(initialSize,
+ AsyncDualMemoryLimiter.LimitType.HEAP_MEMORY, isPermitRequestCancelled, initialPermits -> {
Review Comment:
> Just add a tip: if the data does not exist in the cache, it may query ZK, which will also use direct memory
I agree that there might be other sources of memory allocations. One example is AbstractMetadataStore's childrenCache which consumes heap memory without bounds. #24868 addresses that. In the case of ZK, direct memory usage itself is fairly low.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "poorbarcode (via GitHub)" <gi...@apache.org>.
poorbarcode commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2476939392
##########
pulsar-common/src/main/java/org/apache/pulsar/common/semaphore/AsyncSemaphoreImpl.java:
##########
@@ -0,0 +1,330 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.common.semaphore;
+
+import io.netty.util.concurrent.DefaultThreadFactory;
+import java.util.Queue;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicLongFieldUpdater;
+import java.util.function.BooleanSupplier;
+import java.util.function.LongConsumer;
+import org.apache.pulsar.common.util.Runnables;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of AsyncSemaphore with timeout and queue size limits.
+ */
+public class AsyncSemaphoreImpl implements AsyncSemaphore, AutoCloseable {
+ private static final Logger log = LoggerFactory.getLogger(AsyncSemaphoreImpl.class);
+
+ private final AtomicLong availablePermits;
+ private final Queue<PendingRequest> queue;
+ private final long maxPermits;
+ private final long timeoutMillis;
+ private final ScheduledExecutorService executor;
+ private final boolean shutdownExecutor;
+ private final LongConsumer queueLatencyRecorder;
+ private final AtomicBoolean closed = new AtomicBoolean(false);
+ private final Runnable processQueueRunnable = Runnables.catchingAndLoggingThrowables(this::internalProcessQueue);
+
+ /**
+ * Creates an AsyncSemaphoreImpl with the given parameters.
+ * @param maxPermits max number of permits available for acquisition, set to <= 0 for unbounded semaphore (not
+ * recommended)
+ * @param maxQueueSize max number of requests that can be queued, set to <= 0 for unbounded queue (not recommended)
+ * @param timeoutMillis timeout in milliseconds for acquiring permits
+ */
+ public AsyncSemaphoreImpl(long maxPermits, int maxQueueSize, long timeoutMillis) {
+ this(maxPermits, maxQueueSize, timeoutMillis, maxPermits > 0 ? createExecutor() : null, maxPermits > 0, null);
+ }
+
+ /**
+ * Creates an AsyncSemaphoreImpl with the given parameters.
+ * @param maxPermits max number of permits available for acquisition, set to <= 0 for unbounded semaphore (not
+ * recommended)
+ * @param maxQueueSize max number of requests that can be queued, set to <= 0 for unbounded queue (not recommended)
+ * @param timeoutMillis timeout in milliseconds for acquiring permits
+ * @param executor executor service to use for scheduling timeouts, it is expected to be single threaded
+ * @param queueLatencyRecorder consumer to record queue latency, Long.MAX_VALUE is used for requests that timed out
+ */
+ public AsyncSemaphoreImpl(long maxPermits, int maxQueueSize, long timeoutMillis,
+ ScheduledExecutorService executor, LongConsumer queueLatencyRecorder) {
+ this(maxPermits, maxQueueSize, timeoutMillis, executor, false, queueLatencyRecorder);
+ }
+
+ AsyncSemaphoreImpl(long maxPermits, int maxQueueSize, long timeoutMillis, ScheduledExecutorService executor,
+ boolean shutdownExecutor, LongConsumer queueLatencyRecorder) {
+ this.availablePermits = new AtomicLong(maxPermits);
+ this.maxPermits = maxPermits;
+ this.queue = maxQueueSize > 0 ? new ArrayBlockingQueue<>(maxQueueSize) : new LinkedBlockingQueue<>();
+ this.timeoutMillis = timeoutMillis;
+ this.executor = executor;
+ this.shutdownExecutor = shutdownExecutor;
+ this.queueLatencyRecorder = queueLatencyRecorder;
+ }
+
+ private static ScheduledExecutorService createExecutor() {
+ return Executors.newSingleThreadScheduledExecutor(
+ new DefaultThreadFactory("async-semaphore-executor"));
+ }
+
+ @Override
+ public CompletableFuture<AsyncSemaphorePermit> acquire(long permits, BooleanSupplier isCancelled) {
+ return internalAcquire(permits, permits, isCancelled);
+ }
+
+ private CompletableFuture<AsyncSemaphorePermit> internalAcquire(long permits, long acquirePermits,
+ BooleanSupplier isCancelled) {
+ validatePermits(permits);
+
+ // if maximum permits is <= 0, then the semaphore is unbounded
+ if (isUnbounded()) {
+ return CompletableFuture.completedFuture(new SemaphorePermit(permits));
+ }
+
+ CompletableFuture<AsyncSemaphorePermit> future = new CompletableFuture<>();
+
+ if (closed.get()) {
+ future.completeExceptionally(new PermitAcquireAlreadyClosedException("Semaphore is closed"));
+ return future;
+ }
+
+ PendingRequest request = new PendingRequest(permits, acquirePermits, future, isCancelled);
+ if (!queue.offer(request)) {
+ future.completeExceptionally(new PermitAcquireQueueFullException(
+ "Semaphore queue is full"));
+ return future;
+ }
+ // Schedule timeout
+ ScheduledFuture<?> timeoutTask = executor.schedule(() -> {
+ if (!request.future.isDone() && queue.remove(request)) {
+ // timeout is recorded with Long.MAX_VALUE as the age
+ recordQueueLatency(Long.MAX_VALUE);
+ // also record the time in the queue
+ recordQueueLatency(request.getAgeNanos());
+ future.completeExceptionally(new PermitAcquireTimeoutException(
+ "Permit acquisition timed out"));
+ // the next request might have smaller permits and that might be processed
+ processQueue();
+ }
+ }, timeoutMillis, TimeUnit.MILLISECONDS);
+ request.setTimeoutTask(timeoutTask);
+
+ processQueue();
+ return future;
+ }
+
+ private void validatePermits(long permits) {
+ if (permits < 0) {
+ throw new IllegalArgumentException("Invalid negative permits value: " + permits);
+ }
+ if (!isUnbounded() && permits > maxPermits) {
+ throw new IllegalArgumentException(
+ "Requested permits=" + permits + " is larger than maxPermits=" + maxPermits);
+ }
+ }
+
+ private boolean isUnbounded() {
+ return maxPermits <= 0;
+ }
+
+ private void recordQueueLatency(long ageNanos) {
+ if (queueLatencyRecorder != null) {
+ queueLatencyRecorder.accept(ageNanos);
+ }
+ }
+
+ @Override
+ public CompletableFuture<AsyncSemaphorePermit> update(AsyncSemaphorePermit permit, long newPermits,
+ BooleanSupplier isCancelled) {
+ validatePermits(newPermits);
+ if (isUnbounded()) {
+ return CompletableFuture.completedFuture(new SemaphorePermit(newPermits));
+ }
+ long oldPermits = permit.getPermits();
+ long additionalPermits = newPermits - oldPermits;
+ // mark the old permits as released without adding the permits to availablePermits
+ castToImplementation(permit).releasePermits();
+ if (additionalPermits > 0) {
+ return internalAcquire(newPermits, additionalPermits, isCancelled);
+ } else {
+ // new permits are less than the old ones, so we return the difference
+ availablePermits.addAndGet(-additionalPermits);
+ processQueue();
+ // return the new permits immediately
+ return CompletableFuture.completedFuture(new SemaphorePermit(newPermits));
Review Comment:
There should be a bug:
- `withAcquiredPermits`
- acquired estimated permits ( `100` )
- `withUpdatedPermits`
- update permits when got a real size( `50`)
- release `50` permits here
- finally the permits acquired will be released
- `withAcquiredPermits -> releaser`: releases `100` permits
- `withUpdatedPermits -> releaser`: releases `50` permits
- Issue occurs: the permits is larger than the expected value `100`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "poorbarcode (via GitHub)" <gi...@apache.org>.
poorbarcode commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2476945471
##########
pulsar-common/src/main/java/org/apache/pulsar/common/semaphore/AsyncSemaphoreImpl.java:
##########
@@ -0,0 +1,330 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.common.semaphore;
+
+import io.netty.util.concurrent.DefaultThreadFactory;
+import java.util.Queue;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicLongFieldUpdater;
+import java.util.function.BooleanSupplier;
+import java.util.function.LongConsumer;
+import org.apache.pulsar.common.util.Runnables;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of AsyncSemaphore with timeout and queue size limits.
+ */
+public class AsyncSemaphoreImpl implements AsyncSemaphore, AutoCloseable {
+ private static final Logger log = LoggerFactory.getLogger(AsyncSemaphoreImpl.class);
+
+ private final AtomicLong availablePermits;
+ private final Queue<PendingRequest> queue;
+ private final long maxPermits;
+ private final long timeoutMillis;
+ private final ScheduledExecutorService executor;
+ private final boolean shutdownExecutor;
+ private final LongConsumer queueLatencyRecorder;
+ private final AtomicBoolean closed = new AtomicBoolean(false);
+ private final Runnable processQueueRunnable = Runnables.catchingAndLoggingThrowables(this::internalProcessQueue);
+
+ /**
+ * Creates an AsyncSemaphoreImpl with the given parameters.
+ * @param maxPermits max number of permits available for acquisition, set to <= 0 for unbounded semaphore (not
+ * recommended)
+ * @param maxQueueSize max number of requests that can be queued, set to <= 0 for unbounded queue (not recommended)
+ * @param timeoutMillis timeout in milliseconds for acquiring permits
+ */
+ public AsyncSemaphoreImpl(long maxPermits, int maxQueueSize, long timeoutMillis) {
+ this(maxPermits, maxQueueSize, timeoutMillis, maxPermits > 0 ? createExecutor() : null, maxPermits > 0, null);
+ }
+
+ /**
+ * Creates an AsyncSemaphoreImpl with the given parameters.
+ * @param maxPermits max number of permits available for acquisition, set to <= 0 for unbounded semaphore (not
+ * recommended)
+ * @param maxQueueSize max number of requests that can be queued, set to <= 0 for unbounded queue (not recommended)
+ * @param timeoutMillis timeout in milliseconds for acquiring permits
+ * @param executor executor service to use for scheduling timeouts, it is expected to be single threaded
+ * @param queueLatencyRecorder consumer to record queue latency, Long.MAX_VALUE is used for requests that timed out
+ */
+ public AsyncSemaphoreImpl(long maxPermits, int maxQueueSize, long timeoutMillis,
+ ScheduledExecutorService executor, LongConsumer queueLatencyRecorder) {
+ this(maxPermits, maxQueueSize, timeoutMillis, executor, false, queueLatencyRecorder);
+ }
+
+ AsyncSemaphoreImpl(long maxPermits, int maxQueueSize, long timeoutMillis, ScheduledExecutorService executor,
+ boolean shutdownExecutor, LongConsumer queueLatencyRecorder) {
+ this.availablePermits = new AtomicLong(maxPermits);
+ this.maxPermits = maxPermits;
+ this.queue = maxQueueSize > 0 ? new ArrayBlockingQueue<>(maxQueueSize) : new LinkedBlockingQueue<>();
+ this.timeoutMillis = timeoutMillis;
+ this.executor = executor;
+ this.shutdownExecutor = shutdownExecutor;
+ this.queueLatencyRecorder = queueLatencyRecorder;
+ }
+
+ private static ScheduledExecutorService createExecutor() {
+ return Executors.newSingleThreadScheduledExecutor(
+ new DefaultThreadFactory("async-semaphore-executor"));
+ }
+
+ @Override
+ public CompletableFuture<AsyncSemaphorePermit> acquire(long permits, BooleanSupplier isCancelled) {
+ return internalAcquire(permits, permits, isCancelled);
+ }
+
+ private CompletableFuture<AsyncSemaphorePermit> internalAcquire(long permits, long acquirePermits,
+ BooleanSupplier isCancelled) {
+ validatePermits(permits);
+
+ // if maximum permits is <= 0, then the semaphore is unbounded
+ if (isUnbounded()) {
+ return CompletableFuture.completedFuture(new SemaphorePermit(permits));
+ }
+
+ CompletableFuture<AsyncSemaphorePermit> future = new CompletableFuture<>();
+
+ if (closed.get()) {
+ future.completeExceptionally(new PermitAcquireAlreadyClosedException("Semaphore is closed"));
+ return future;
+ }
+
+ PendingRequest request = new PendingRequest(permits, acquirePermits, future, isCancelled);
+ if (!queue.offer(request)) {
+ future.completeExceptionally(new PermitAcquireQueueFullException(
+ "Semaphore queue is full"));
+ return future;
+ }
+ // Schedule timeout
+ ScheduledFuture<?> timeoutTask = executor.schedule(() -> {
+ if (!request.future.isDone() && queue.remove(request)) {
+ // timeout is recorded with Long.MAX_VALUE as the age
+ recordQueueLatency(Long.MAX_VALUE);
+ // also record the time in the queue
+ recordQueueLatency(request.getAgeNanos());
+ future.completeExceptionally(new PermitAcquireTimeoutException(
+ "Permit acquisition timed out"));
+ // the next request might have smaller permits and that might be processed
+ processQueue();
+ }
+ }, timeoutMillis, TimeUnit.MILLISECONDS);
+ request.setTimeoutTask(timeoutTask);
+
+ processQueue();
+ return future;
+ }
+
+ private void validatePermits(long permits) {
+ if (permits < 0) {
+ throw new IllegalArgumentException("Invalid negative permits value: " + permits);
+ }
+ if (!isUnbounded() && permits > maxPermits) {
+ throw new IllegalArgumentException(
+ "Requested permits=" + permits + " is larger than maxPermits=" + maxPermits);
+ }
+ }
+
+ private boolean isUnbounded() {
+ return maxPermits <= 0;
+ }
+
+ private void recordQueueLatency(long ageNanos) {
+ if (queueLatencyRecorder != null) {
+ queueLatencyRecorder.accept(ageNanos);
+ }
+ }
+
+ @Override
+ public CompletableFuture<AsyncSemaphorePermit> update(AsyncSemaphorePermit permit, long newPermits,
+ BooleanSupplier isCancelled) {
+ validatePermits(newPermits);
+ if (isUnbounded()) {
+ return CompletableFuture.completedFuture(new SemaphorePermit(newPermits));
Review Comment:
https://github.com/apache/pulsar/pull/24833/files#r2476939392 The same bug will happen here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2479071244
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/ServerCnx.java:
##########
@@ -2610,6 +2581,93 @@ protected void handleGetTopicsOfNamespace(CommandGetTopicsOfNamespace commandGet
}
}
+ private void internalHandleGetTopicsOfNamespace(String namespace, NamespaceName namespaceName, long requestId,
+ CommandGetTopicsOfNamespace.Mode mode,
+ Optional<String> topicsPattern, Optional<String> topicsHash,
+ Semaphore lookupSemaphore) {
+ BooleanSupplier isPermitRequestCancelled = () -> !ctx().channel().isActive();
+ TopicListSizeResultCache.ResultHolder
+ listSizeHolder = service.getTopicListSizeResultCache().getTopicListSize(namespaceName.toString(), mode);
+ listSizeHolder.getSizeAsync().thenAccept(initialSize -> {
+ maxTopicListInFlightLimiter.withAcquiredPermits(initialSize,
+ AsyncDualMemoryLimiter.LimitType.HEAP_MEMORY, isPermitRequestCancelled, initialPermits -> {
+ return getBrokerService().pulsar().getNamespaceService()
+ .getListOfUserTopics(namespaceName, mode)
+ .thenAccept(topics -> {
+ long actualSize = TopicListMemoryLimiter.estimateTopicListSize(topics);
+ listSizeHolder.updateSize(actualSize);
+ maxTopicListInFlightLimiter.withUpdatedPermits(initialPermits, actualSize,
Review Comment:
> No, the implementation is like follows, which never covered the case I said above
>
> * update
> * call `internalAcquire` if `realSize > estimatedSize`, see also https://github.com/apache/pulsar/pull/24833/files#diff-1daf74806e4d1f752d9576bd63d71678a7a46cbc7a1cf56f71ba88719f87d346R173-R175
> * `internalAcquire` will pending the current thread or throws errors, see also https://github.com/apache/pulsar/pull/24833/files#diff-1daf74806e4d1f752d9576bd63d71678a7a46cbc7a1cf56f71ba88719f87d346R117-R122
I don't know exactly what you are referring to. I made some modifications so that permits on the original instance are only released after successfully acquiring the new permits. If you have something else in mind, it could be useful to reveal more details.
> Instead of updating and acquiring additional permits if the actual size is larger than estimated, we'd better improve as follows:
This is intentionally designed in the way it is. I'll explain in the following paragraph.
> * The current request does not need to acquire permits again; only updating the permits that were borrowed out is enough, even though the permits were over-acquired
In the current design the original permit instance is reseted when the permits are moved to a new instance successfully (that was now fixed). This is intended behavior so that any later logic can always release the original permit without side-effects. This makes it easier to prevent permits leaks when such pattern is applied. For example, the org.apache.pulsar.common.semaphore.AsyncDualMemoryLimiterUtil#withPermitsFuture and org.apache.pulsar.common.semaphore.AsyncDualMemoryLimiterUtil#acquireDirectMemoryPermitsAndWriteAndFlush helpers are simpler to implement when there's 2 separate permit instances. You can also review the implementation code and see why I came to this design decision.
> * Updates the permits that were borrowed out, to limit the following permits acquisition;
I wonder if you meant to point out the bug that there was previously where original permits were released before the new permit (with updated amount) was successfully acquired?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "poorbarcode (via GitHub)" <gi...@apache.org>.
poorbarcode commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2492539745
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/TopicListService.java:
##########
@@ -101,6 +112,7 @@ public void accept(String topicName, NotificationType notificationType) {
private final boolean enableSubscriptionPatternEvaluation;
private final int maxSubscriptionPatternLength;
private final ConcurrentLongHashMap<CompletableFuture<TopicListWatcher>> watchers;
+ private final Backoff retryBackoff;
public TopicListService(PulsarService pulsar, ServerCnx connection,
Review Comment:
> unless it times out or the queue fills up
That situation is what I want to say. It will cause consumers to be unable to receive messages that were sent to the new partitions until the client application restarts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari merged PR #24833:
URL: https://github.com/apache/pulsar/pull/24833
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "poorbarcode (via GitHub)" <gi...@apache.org>.
poorbarcode commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2487233605
##########
pulsar-common/src/main/java/org/apache/pulsar/common/semaphore/AsyncSemaphoreImpl.java:
##########
@@ -0,0 +1,330 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.common.semaphore;
+
+import io.netty.util.concurrent.DefaultThreadFactory;
+import java.util.Queue;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicLongFieldUpdater;
+import java.util.function.BooleanSupplier;
+import java.util.function.LongConsumer;
+import org.apache.pulsar.common.util.Runnables;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of AsyncSemaphore with timeout and queue size limits.
+ */
+public class AsyncSemaphoreImpl implements AsyncSemaphore, AutoCloseable {
+ private static final Logger log = LoggerFactory.getLogger(AsyncSemaphoreImpl.class);
+
+ private final AtomicLong availablePermits;
+ private final Queue<PendingRequest> queue;
+ private final long maxPermits;
+ private final long timeoutMillis;
+ private final ScheduledExecutorService executor;
+ private final boolean shutdownExecutor;
+ private final LongConsumer queueLatencyRecorder;
+ private final AtomicBoolean closed = new AtomicBoolean(false);
+ private final Runnable processQueueRunnable = Runnables.catchingAndLoggingThrowables(this::internalProcessQueue);
+
+ /**
+ * Creates an AsyncSemaphoreImpl with the given parameters.
+ * @param maxPermits max number of permits available for acquisition, set to <= 0 for unbounded semaphore (not
+ * recommended)
+ * @param maxQueueSize max number of requests that can be queued, set to <= 0 for unbounded queue (not recommended)
+ * @param timeoutMillis timeout in milliseconds for acquiring permits
+ */
+ public AsyncSemaphoreImpl(long maxPermits, int maxQueueSize, long timeoutMillis) {
+ this(maxPermits, maxQueueSize, timeoutMillis, maxPermits > 0 ? createExecutor() : null, maxPermits > 0, null);
+ }
+
+ /**
+ * Creates an AsyncSemaphoreImpl with the given parameters.
+ * @param maxPermits max number of permits available for acquisition, set to <= 0 for unbounded semaphore (not
+ * recommended)
+ * @param maxQueueSize max number of requests that can be queued, set to <= 0 for unbounded queue (not recommended)
+ * @param timeoutMillis timeout in milliseconds for acquiring permits
+ * @param executor executor service to use for scheduling timeouts, it is expected to be single threaded
+ * @param queueLatencyRecorder consumer to record queue latency, Long.MAX_VALUE is used for requests that timed out
+ */
+ public AsyncSemaphoreImpl(long maxPermits, int maxQueueSize, long timeoutMillis,
+ ScheduledExecutorService executor, LongConsumer queueLatencyRecorder) {
+ this(maxPermits, maxQueueSize, timeoutMillis, executor, false, queueLatencyRecorder);
+ }
+
+ AsyncSemaphoreImpl(long maxPermits, int maxQueueSize, long timeoutMillis, ScheduledExecutorService executor,
+ boolean shutdownExecutor, LongConsumer queueLatencyRecorder) {
+ this.availablePermits = new AtomicLong(maxPermits);
+ this.maxPermits = maxPermits;
+ this.queue = maxQueueSize > 0 ? new ArrayBlockingQueue<>(maxQueueSize) : new LinkedBlockingQueue<>();
+ this.timeoutMillis = timeoutMillis;
+ this.executor = executor;
+ this.shutdownExecutor = shutdownExecutor;
+ this.queueLatencyRecorder = queueLatencyRecorder;
+ }
+
+ private static ScheduledExecutorService createExecutor() {
+ return Executors.newSingleThreadScheduledExecutor(
+ new DefaultThreadFactory("async-semaphore-executor"));
+ }
+
+ @Override
+ public CompletableFuture<AsyncSemaphorePermit> acquire(long permits, BooleanSupplier isCancelled) {
+ return internalAcquire(permits, permits, isCancelled);
+ }
+
+ private CompletableFuture<AsyncSemaphorePermit> internalAcquire(long permits, long acquirePermits,
+ BooleanSupplier isCancelled) {
+ validatePermits(permits);
+
+ // if maximum permits is <= 0, then the semaphore is unbounded
+ if (isUnbounded()) {
+ return CompletableFuture.completedFuture(new SemaphorePermit(permits));
+ }
+
+ CompletableFuture<AsyncSemaphorePermit> future = new CompletableFuture<>();
+
+ if (closed.get()) {
+ future.completeExceptionally(new PermitAcquireAlreadyClosedException("Semaphore is closed"));
+ return future;
+ }
+
+ PendingRequest request = new PendingRequest(permits, acquirePermits, future, isCancelled);
+ if (!queue.offer(request)) {
+ future.completeExceptionally(new PermitAcquireQueueFullException(
+ "Semaphore queue is full"));
+ return future;
+ }
+ // Schedule timeout
+ ScheduledFuture<?> timeoutTask = executor.schedule(() -> {
+ if (!request.future.isDone() && queue.remove(request)) {
+ // timeout is recorded with Long.MAX_VALUE as the age
+ recordQueueLatency(Long.MAX_VALUE);
+ // also record the time in the queue
+ recordQueueLatency(request.getAgeNanos());
+ future.completeExceptionally(new PermitAcquireTimeoutException(
+ "Permit acquisition timed out"));
+ // the next request might have smaller permits and that might be processed
+ processQueue();
+ }
+ }, timeoutMillis, TimeUnit.MILLISECONDS);
+ request.setTimeoutTask(timeoutTask);
+
+ processQueue();
+ return future;
+ }
+
+ private void validatePermits(long permits) {
+ if (permits < 0) {
+ throw new IllegalArgumentException("Invalid negative permits value: " + permits);
+ }
+ if (!isUnbounded() && permits > maxPermits) {
+ throw new IllegalArgumentException(
+ "Requested permits=" + permits + " is larger than maxPermits=" + maxPermits);
+ }
+ }
+
+ private boolean isUnbounded() {
+ return maxPermits <= 0;
+ }
+
+ private void recordQueueLatency(long ageNanos) {
+ if (queueLatencyRecorder != null) {
+ queueLatencyRecorder.accept(ageNanos);
+ }
+ }
+
+ @Override
+ public CompletableFuture<AsyncSemaphorePermit> update(AsyncSemaphorePermit permit, long newPermits,
+ BooleanSupplier isCancelled) {
+ validatePermits(newPermits);
+ if (isUnbounded()) {
+ return CompletableFuture.completedFuture(new SemaphorePermit(newPermits));
Review Comment:
Mark the comment as solved with the comment https://github.com/apache/pulsar/pull/24833#discussion_r2487231319
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "Denovo1998 (via GitHub)" <gi...@apache.org>.
Denovo1998 commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2432254493
##########
pulsar-common/src/main/java/org/apache/pulsar/common/semaphore/AsyncSemaphoreImpl.java:
##########
@@ -0,0 +1,286 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.common.semaphore;
+
+import io.netty.util.concurrent.DefaultThreadFactory;
+import java.util.Queue;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicLongFieldUpdater;
+import java.util.function.BooleanSupplier;
+import java.util.function.LongConsumer;
+import org.apache.pulsar.common.util.Runnables;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of AsyncSemaphore with timeout and queue size limits.
+ */
+public class AsyncSemaphoreImpl implements AsyncSemaphore, AutoCloseable {
+ private static final Logger log = LoggerFactory.getLogger(AsyncSemaphoreImpl.class);
+
+ private final AtomicLong availablePermits;
+ private final Queue<PendingRequest> queue;
+ private final long maxPermits;
+ private final long timeoutMillis;
+ private final ScheduledExecutorService executor;
+ private final boolean shutdownExecutor;
+ private final LongConsumer queueLatencyRecorder;
+ private final AtomicBoolean closed = new AtomicBoolean(false);
+ private final Runnable processQueueRunnable = Runnables.catchingAndLoggingThrowables(this::internalProcessQueue);
+
+ public AsyncSemaphoreImpl(long maxPermits, int maxQueueSize, long timeoutMillis) {
+ this(maxPermits, maxQueueSize, timeoutMillis, createExecutor(), true, null);
+ }
+
+ public AsyncSemaphoreImpl(long maxPermits, int maxQueueSize, long timeoutMillis,
+ ScheduledExecutorService executor, LongConsumer queueLatencyRecorder) {
+ this(maxPermits, maxQueueSize, timeoutMillis, executor, false, queueLatencyRecorder);
+ }
+
+ AsyncSemaphoreImpl(long maxPermits, int maxQueueSize, long timeoutMillis, ScheduledExecutorService executor,
+ boolean shutdownExecutor, LongConsumer queueLatencyRecorder) {
+ this.availablePermits = new AtomicLong(maxPermits);
+ this.maxPermits = maxPermits;
+ this.queue = new ArrayBlockingQueue<>(maxQueueSize);
+ this.timeoutMillis = timeoutMillis;
+ this.executor = executor;
+ this.shutdownExecutor = shutdownExecutor;
+ this.queueLatencyRecorder = queueLatencyRecorder;
+ }
+
+ private static ScheduledExecutorService createExecutor() {
+ return Executors.newSingleThreadScheduledExecutor(
+ new DefaultThreadFactory("async-semaphore-executor"));
+ }
+
+ @Override
+ public CompletableFuture<AsyncSemaphorePermit> acquire(long permits, BooleanSupplier isCancelled) {
+ return internalAcquire(permits, permits, isCancelled);
+ }
+
+ private CompletableFuture<AsyncSemaphorePermit> internalAcquire(long permits, long acquirePermits,
+ BooleanSupplier isCancelled) {
+ if (permits < 0) {
+ throw new IllegalArgumentException("Invalid permits value: " + permits);
+ }
+
+ CompletableFuture<AsyncSemaphorePermit> future = new CompletableFuture<>();
+
+ if (closed.get()) {
+ future.completeExceptionally(new PermitAcquireAlreadyClosedException("Semaphore is closed"));
+ return future;
+ }
+
+ PendingRequest request = new PendingRequest(permits, acquirePermits, future, isCancelled);
+ if (!queue.offer(request)) {
+ future.completeExceptionally(new PermitAcquireQueueFullException(
+ "Semaphore queue is full"));
+ return future;
+ }
+ // Schedule timeout
+ ScheduledFuture<?> timeoutTask = executor.schedule(() -> {
+ if (!request.future.isDone() && queue.remove(request)) {
+ // timeout is recorded with Long.MAX_VALUE as the age
+ recordQueueLatency(Long.MAX_VALUE);
+ // also record the time in the queue
+ recordQueueLatency(request.getAgeNanos());
+ future.completeExceptionally(new PermitAcquireTimeoutException(
+ "Permit acquisition timed out"));
+ // the next request might have smaller permits and that might be processed
+ processQueue();
+ }
+ }, timeoutMillis, TimeUnit.MILLISECONDS);
+ request.setTimeoutTask(timeoutTask);
+
+ processQueue();
+ return future;
+ }
+
+ private void recordQueueLatency(long ageNanos) {
+ if (queueLatencyRecorder != null) {
+ queueLatencyRecorder.accept(ageNanos);
+ }
+ }
+
+ @Override
+ public CompletableFuture<AsyncSemaphorePermit> update(AsyncSemaphorePermit permit, long newPermits,
+ BooleanSupplier isCancelled) {
+ if (newPermits < 0) {
+ throw new IllegalArgumentException("Invalid permits value: " + newPermits);
+ }
+ long oldPermits = permit.getPermits();
+ long additionalPermits = newPermits - oldPermits;
+ // mark the old permits as released without adding the permits to availablePermits
+ castToImplementation(permit).releasePermits();
+ if (additionalPermits > 0) {
+ return internalAcquire(newPermits, additionalPermits, isCancelled);
+ } else {
+ // new permits are less than the old ones, so we return the difference
+ availablePermits.addAndGet(-additionalPermits);
+ processQueue();
+ // return the new permits immediately
+ return CompletableFuture.completedFuture(new SemaphorePermit(newPermits));
+ }
+ }
+
+ @Override
+ public void release(AsyncSemaphorePermit permit) {
+ availablePermits.addAndGet(castToImplementation(permit).releasePermits());
+ processQueue();
+ }
+
+ @Override
+ public long getAvailablePermits() {
+ return availablePermits.get();
+ }
+
+ @Override
+ public long getAcquiredPermits() {
+ return maxPermits - availablePermits.get();
+ }
+
+ @Override
+ public int getQueueSize() {
+ return queue.size();
+ }
+
+ private SemaphorePermit castToImplementation(AsyncSemaphorePermit permit) {
+ if (permit instanceof SemaphorePermit semaphorePermit) {
+ return semaphorePermit;
+ } else {
+ throw new IllegalArgumentException("Invalid permit type");
+ }
+ }
+
+ private void processQueue() {
+ if (closed.get()) {
+ return;
+ }
+ executor.execute(processQueueRunnable);
+ }
+
+ private void internalProcessQueue() {
Review Comment:
Oh! I understand. This does indeed need to be recorded.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#issuecomment-3408521407
To properly handle a large amount of requests, it was necessary to add a solution for keeping a cache of the estimated size of the topic listing. Without the solution heap memory would run out in the ProxyPatternConsumerBackPressureMultipleConsumersTest test after increasing concurrency. While debugging, I noticed that this happened due to the low initial permits. Since a lot of inflight requests were accepted, it caused heap memory to grow very quickly. The solution is explained [in the javadoc of TopicListSizeResultCache](https://github.com/lhotari/pulsar/blob/lh-implement-PIP-442/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/topiclistlimit/TopicListSizeResultCache.java). It lets the first request to proceed with the initial permits and concurrent requests will wait until the first request updates the topic list size. This will ensure that all subsequent requests are using a sane value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2426526090
##########
org/apache/pulsar/common/util/memory/AsyncSemaphore.java:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.common.util.memory;
+
+import java.util.concurrent.CompletableFuture;
+
+/**
+ * Interface for asynchronous semaphore operations.
+ * Used for managing concurrent access to limited resources.
+ */
+public interface AsyncSemaphore {
Review Comment:
No, what do you mean?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "poorbarcode (via GitHub)" <gi...@apache.org>.
poorbarcode commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2476710868
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/ServerCnx.java:
##########
@@ -2610,6 +2581,93 @@ protected void handleGetTopicsOfNamespace(CommandGetTopicsOfNamespace commandGet
}
}
+ private void internalHandleGetTopicsOfNamespace(String namespace, NamespaceName namespaceName, long requestId,
+ CommandGetTopicsOfNamespace.Mode mode,
+ Optional<String> topicsPattern, Optional<String> topicsHash,
+ Semaphore lookupSemaphore) {
+ BooleanSupplier isPermitRequestCancelled = () -> !ctx().channel().isActive();
+ TopicListSizeResultCache.ResultHolder
+ listSizeHolder = service.getTopicListSizeResultCache().getTopicListSize(namespaceName.toString(), mode);
+ listSizeHolder.getSizeAsync().thenAccept(initialSize -> {
+ maxTopicListInFlightLimiter.withAcquiredPermits(initialSize,
+ AsyncDualMemoryLimiter.LimitType.HEAP_MEMORY, isPermitRequestCancelled, initialPermits -> {
+ return getBrokerService().pulsar().getNamespaceService()
+ .getListOfUserTopics(namespaceName, mode)
+ .thenAccept(topics -> {
+ long actualSize = TopicListMemoryLimiter.estimateTopicListSize(topics);
+ listSizeHolder.updateSize(actualSize);
+ maxTopicListInFlightLimiter.withUpdatedPermits(initialPermits, actualSize,
Review Comment:
No, the implementation is like follows
- update
- call `internalAcquire` if `realSize > estimatedSize`, see also https://github.com/apache/pulsar/pull/24833/files#diff-1daf74806e4d1f752d9576bd63d71678a7a46cbc7a1cf56f71ba88719f87d346R173-R175
- `internalAcquire` will pending the current thread or throws errors, see also https://github.com/apache/pulsar/pull/24833/files#diff-1daf74806e4d1f752d9576bd63d71678a7a46cbc7a1cf56f71ba88719f87d346R117-R122
- **method update**
```java
public CompletableFuture<AsyncSemaphorePermit> update(AsyncSemaphorePermit permit, long newPermits,
BooleanSupplier isCancelled) {
long oldPermits = permit.getPermits();
long additionalPermits = newPermits - oldPermits;
if (additionalPermits > 0) {
return internalAcquire(newPermits, additionalPermits, isCancelled);
}
}
```
- **pending requests**
```java
private CompletableFuture<AsyncSemaphorePermit> internalAcquire(long permits, long acquirePermits,
BooleanSupplier isCancelled) {
validatePermits(permits);
PendingRequest request = new PendingRequest(permits, acquirePermits, future, isCancelled);
if (!queue.offer(request)) {
future.completeExceptionally(new PermitAcquireQueueFullException(
"Semaphore queue is full"));
return future;
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "poorbarcode (via GitHub)" <gi...@apache.org>.
poorbarcode commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2476958091
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/TopicListService.java:
##########
@@ -101,6 +112,7 @@ public void accept(String topicName, NotificationType notificationType) {
private final boolean enableSubscriptionPatternEvaluation;
private final int maxSubscriptionPatternLength;
private final ConcurrentLongHashMap<CompletableFuture<TopicListWatcher>> watchers;
+ private final Backoff retryBackoff;
public TopicListService(PulsarService pulsar, ServerCnx connection,
Review Comment:
Please don't close this comment for now. I will review this piece of code again. I will solve it as I completed the review
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/TopicListService.java:
##########
@@ -101,6 +112,7 @@ public void accept(String topicName, NotificationType notificationType) {
private final boolean enableSubscriptionPatternEvaluation;
private final int maxSubscriptionPatternLength;
private final ConcurrentLongHashMap<CompletableFuture<TopicListWatcher>> watchers;
+ private final Backoff retryBackoff;
public TopicListService(PulsarService pulsar, ServerCnx connection,
Review Comment:
Please don't close this comment for now. I will review this piece of code again. I will solve it as I have completed the review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "codelipenghui (via GitHub)" <gi...@apache.org>.
codelipenghui commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2436732605
##########
pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/LookupProxyHandler.java:
##########
@@ -357,6 +359,76 @@ private void performGetTopicsOfNamespace(long clientRequestId,
});
}
+ private void internalPerformGetTopicsOfNamespace(long clientRequestId, String namespaceName,
+ CommandGetTopicsOfNamespace.Mode mode, ClientCnx clientCnx,
+ ByteBuf command, long requestId) {
+ BooleanSupplier isPermitRequestCancelled = () -> !proxyConnection.ctx().channel().isActive();
+ TopicListSizeResultCache.ResultHolder
+ listSizeHolder = topicListSizeResultCache.getTopicListSize(namespaceName, mode);
+ listSizeHolder.getSizeAsync().thenAccept(initialSize -> {
+ maxTopicListInFlightLimiter.withAcquiredPermits(initialSize,
+ AsyncDualMemoryLimiter.LimitType.HEAP_MEMORY, isPermitRequestCancelled, initialPermits -> {
+ return clientCnx.newGetTopicsOfNamespace(command, requestId).whenComplete((r, t) -> {
+ if (t != null) {
+ log.warn("[{}] Failed to get TopicsOfNamespace {}: {}", clientAddress, namespaceName,
+ t.getMessage());
+ listSizeHolder.resetIfInitializing();
+ writeAndFlush(Commands.newError(clientRequestId, getServerError(t), t.getMessage()));
+ } else {
+ long actualSize =
+ r.getNonPartitionedOrPartitionTopics().stream()
+ .mapToInt(ByteBufUtil::utf8Bytes) // convert character count to bytes
+ .map(n -> n + 32) // add 32 bytes overhead for each entry
Review Comment:
Same as the above comment
##########
pulsar-broker-common/src/main/java/org/apache/pulsar/broker/topiclistlimit/TopicListMemoryLimiter.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.topiclistlimit;
+
+import io.opentelemetry.api.metrics.DoubleGauge;
+import io.opentelemetry.api.metrics.DoubleHistogram;
+import io.opentelemetry.api.metrics.LongCounter;
+import io.opentelemetry.api.metrics.LongGauge;
+import io.opentelemetry.api.metrics.Meter;
+import io.opentelemetry.api.metrics.ObservableDoubleGauge;
+import io.opentelemetry.api.metrics.ObservableLongUpDownCounter;
+import io.prometheus.client.Collector;
+import io.prometheus.client.CollectorRegistry;
+import io.prometheus.client.Counter;
+import io.prometheus.client.Gauge;
+import io.prometheus.client.Summary;
+import java.util.concurrent.TimeUnit;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.common.semaphore.AsyncDualMemoryLimiterImpl;
+import org.apache.pulsar.common.semaphore.AsyncSemaphore;
+
+/**
+ * Topic list memory limiter that exposes both Prometheus metrics and OpenTelemetry metrics.
+ */
+@Slf4j
+public class TopicListMemoryLimiter extends AsyncDualMemoryLimiterImpl {
+ private final CollectorRegistry collectorRegistry;
+ private final Gauge heapMemoryUsedBytes;
+ private final Gauge heapMemoryLimitBytes;
+ private final Gauge directMemoryUsedBytes;
+ private final Gauge directMemoryLimitBytes;
+ private final Gauge heapQueueSize;
+ private final Gauge heapQueueMaxSize;
+ private final Gauge directQueueSize;
+ private final Gauge directQueueMaxSize;
+ private final Summary heapWaitTimeMs;
+ private final Summary directWaitTimeMs;
+ private final Counter heapTimeoutTotal;
+ private final Counter directTimeoutTotal;
+ private final ObservableDoubleGauge otelHeapMemoryUsedGauge;
+ private final DoubleGauge otelHeapMemoryLimitGauge;
+ private final ObservableDoubleGauge otelDirectMemoryUsedGauge;
+ private final DoubleGauge otelDirectMemoryLimitGauge;
+ private final ObservableLongUpDownCounter otelHeapQueueSize;
+ private final ObservableLongUpDownCounter otelDirectQueueSize;
+ private final DoubleHistogram otelHeapWaitTime;
+ private final DoubleHistogram otelDirectWaitTime;
+ private final LongCounter otelHeapTimeoutTotal;
+ private final LongCounter otelDirectTimeoutTotal;
+
+ public TopicListMemoryLimiter(CollectorRegistry collectorRegistry, String prometheusPrefix,
+ Meter openTelemetryMeter,
+ long maxHeapMemory, int maxHeapQueueSize,
+ long heapTimeoutMillis, long maxDirectMemory, int maxDirectQueueSize,
+ long directTimeoutMillis) {
+ super(maxHeapMemory, maxHeapQueueSize, heapTimeoutMillis, maxDirectMemory, maxDirectQueueSize,
+ directTimeoutMillis);
+ this.collectorRegistry = collectorRegistry;
+
+ AsyncSemaphore heapMemoryLimiter = getLimiter(LimitType.HEAP_MEMORY);
+ AsyncSemaphore directMemoryLimiter = getLimiter(LimitType.DIRECT_MEMORY);
+
+ this.heapMemoryUsedBytes = register(Gauge.build(prometheusPrefix + "topic_list_heap_memory_used_bytes",
+ "Current heap memory used by topic listings")
+ .create()
+ .setChild(new Gauge.Child() {
+ @Override
+ public double get() {
+ return heapMemoryLimiter.getAcquiredPermits();
+ }
+ }));
+ this.otelHeapMemoryUsedGauge = openTelemetryMeter.gaugeBuilder("topic.list.heap.memory.used")
+ .setUnit("By")
+ .setDescription("Current heap memory used by topic listings")
+ .buildWithCallback(observableDoubleMeasurement -> {
+ observableDoubleMeasurement.record(heapMemoryLimiter.getAcquiredPermits());
+ });
+
+ this.heapMemoryLimitBytes = register(Gauge.build(prometheusPrefix + "topic_list_heap_memory_limit_bytes",
+ "Configured heap memory limit")
+ .create());
+ this.heapMemoryLimitBytes.set(maxHeapMemory);
+ this.otelHeapMemoryLimitGauge = openTelemetryMeter.gaugeBuilder("topic.list.heap.memory.limit")
+ .setUnit("By")
+ .setDescription("Configured heap memory limit")
+ .build();
+ this.otelHeapMemoryLimitGauge.set(maxHeapMemory);
+
+ this.directMemoryUsedBytes = register(Gauge.build(prometheusPrefix + "topic_list_direct_memory_used_bytes",
+ "Current direct memory used by topic listings")
+ .create()
+ .setChild(new Gauge.Child() {
+ @Override
+ public double get() {
+ return directMemoryLimiter.getAcquiredPermits();
+ }
+ }));
+ this.otelDirectMemoryUsedGauge = openTelemetryMeter.gaugeBuilder("topic.list.direct.memory.used")
+ .setUnit("By")
+ .setDescription("Current direct memory used by topic listings")
+ .buildWithCallback(observableDoubleMeasurement -> {
+ observableDoubleMeasurement.record(directMemoryLimiter.getAcquiredPermits());
+ });
+
+ this.directMemoryLimitBytes = register(Gauge.build(prometheusPrefix + "topic_list_direct_memory_limit_bytes",
+ "Configured direct memory limit")
+ .create());
+ this.directMemoryLimitBytes.set(maxDirectMemory);
+ this.otelDirectMemoryLimitGauge = openTelemetryMeter.gaugeBuilder("topic.list.direct.memory.limit")
+ .setUnit("By")
+ .setDescription("Configured direct memory limit")
+ .build();
+ this.otelDirectMemoryLimitGauge.set(maxDirectMemory);
+
+ this.heapQueueSize = register(Gauge.build(prometheusPrefix + "topic_list_heap_queue_size",
+ "Current heap memory limiter queue size")
+ .create()
+ .setChild(new Gauge.Child() {
+ @Override
+ public double get() {
+ return heapMemoryLimiter.getQueueSize();
+ }
+ }));
+ this.otelHeapQueueSize = openTelemetryMeter
+ .upDownCounterBuilder("topic.list.heap.queue.size")
+ .setDescription("Current heap memory limiter queue size")
+ .setUnit("1")
+ .buildWithCallback(observableLongMeasurement -> {
+ observableLongMeasurement.record(heapMemoryLimiter.getQueueSize());
+ });
+
+ this.heapQueueMaxSize = register(Gauge.build(prometheusPrefix + "topic_list_heap_queue_max_size",
+ "Maximum heap memory limiter queue size")
+ .create());
+ this.heapQueueMaxSize.set(maxHeapQueueSize);
+ LongGauge otelHeapQueueMaxSize = openTelemetryMeter
+ .gaugeBuilder("topic.list.heap.queue.max.size")
+ .setDescription("Maximum heap memory limiter queue size")
+ .setUnit("1")
+ .ofLongs()
+ .build();
+ otelHeapQueueMaxSize.set(maxHeapQueueSize);
+
+ this.directQueueSize = register(Gauge.build(prometheusPrefix + "topic_list_direct_queue_size",
+ "Current direct memory limiter queue size")
+ .create()
+ .setChild(new Gauge.Child() {
+ @Override
+ public double get() {
+ return directMemoryLimiter.getQueueSize();
+ }
+ }));
+ this.otelDirectQueueSize = openTelemetryMeter
+ .upDownCounterBuilder("topic.list.direct.queue.size")
+ .setDescription("Current direct memory limiter queue size")
+ .setUnit("1")
+ .buildWithCallback(observableLongMeasurement -> {
+ observableLongMeasurement.record(directMemoryLimiter.getQueueSize());
+ });
+
+ this.directQueueMaxSize = register(Gauge.build(prometheusPrefix + "topic_list_direct_queue_max_size",
+ "Maximum direct memory limiter queue size")
+ .create());
+ this.directQueueMaxSize.set(maxDirectQueueSize);
+ LongGauge otelDirectQueueMaxSize = openTelemetryMeter
+ .gaugeBuilder("topic.list.direct.queue.max.size")
+ .setDescription("Maximum direct memory limiter queue size")
+ .setUnit("1")
+ .ofLongs()
+ .build();
+ otelDirectQueueMaxSize.set(maxDirectQueueSize);
+
+ this.heapWaitTimeMs = register(Summary.build(prometheusPrefix + "topic_list_heap_wait_time_ms",
Review Comment:
Based on https://github.com/apache/pulsar/pull/21080, we should change to use a histogram instead of a summary.
##########
pulsar-broker-common/src/main/java/org/apache/pulsar/broker/topiclistlimit/TopicListMemoryLimiter.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.topiclistlimit;
+
+import io.opentelemetry.api.metrics.DoubleGauge;
+import io.opentelemetry.api.metrics.DoubleHistogram;
+import io.opentelemetry.api.metrics.LongCounter;
+import io.opentelemetry.api.metrics.LongGauge;
+import io.opentelemetry.api.metrics.Meter;
+import io.opentelemetry.api.metrics.ObservableDoubleGauge;
+import io.opentelemetry.api.metrics.ObservableLongUpDownCounter;
+import io.prometheus.client.Collector;
+import io.prometheus.client.CollectorRegistry;
+import io.prometheus.client.Counter;
+import io.prometheus.client.Gauge;
+import io.prometheus.client.Summary;
+import java.util.concurrent.TimeUnit;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.common.semaphore.AsyncDualMemoryLimiterImpl;
+import org.apache.pulsar.common.semaphore.AsyncSemaphore;
+
+/**
+ * Topic list memory limiter that exposes both Prometheus metrics and OpenTelemetry metrics.
+ */
+@Slf4j
+public class TopicListMemoryLimiter extends AsyncDualMemoryLimiterImpl {
+ private final CollectorRegistry collectorRegistry;
+ private final Gauge heapMemoryUsedBytes;
+ private final Gauge heapMemoryLimitBytes;
+ private final Gauge directMemoryUsedBytes;
+ private final Gauge directMemoryLimitBytes;
+ private final Gauge heapQueueSize;
+ private final Gauge heapQueueMaxSize;
+ private final Gauge directQueueSize;
+ private final Gauge directQueueMaxSize;
+ private final Summary heapWaitTimeMs;
+ private final Summary directWaitTimeMs;
+ private final Counter heapTimeoutTotal;
+ private final Counter directTimeoutTotal;
+ private final ObservableDoubleGauge otelHeapMemoryUsedGauge;
+ private final DoubleGauge otelHeapMemoryLimitGauge;
+ private final ObservableDoubleGauge otelDirectMemoryUsedGauge;
+ private final DoubleGauge otelDirectMemoryLimitGauge;
+ private final ObservableLongUpDownCounter otelHeapQueueSize;
+ private final ObservableLongUpDownCounter otelDirectQueueSize;
+ private final DoubleHistogram otelHeapWaitTime;
+ private final DoubleHistogram otelDirectWaitTime;
+ private final LongCounter otelHeapTimeoutTotal;
+ private final LongCounter otelDirectTimeoutTotal;
+
+ public TopicListMemoryLimiter(CollectorRegistry collectorRegistry, String prometheusPrefix,
+ Meter openTelemetryMeter,
+ long maxHeapMemory, int maxHeapQueueSize,
+ long heapTimeoutMillis, long maxDirectMemory, int maxDirectQueueSize,
+ long directTimeoutMillis) {
+ super(maxHeapMemory, maxHeapQueueSize, heapTimeoutMillis, maxDirectMemory, maxDirectQueueSize,
+ directTimeoutMillis);
+ this.collectorRegistry = collectorRegistry;
+
+ AsyncSemaphore heapMemoryLimiter = getLimiter(LimitType.HEAP_MEMORY);
+ AsyncSemaphore directMemoryLimiter = getLimiter(LimitType.DIRECT_MEMORY);
+
+ this.heapMemoryUsedBytes = register(Gauge.build(prometheusPrefix + "topic_list_heap_memory_used_bytes",
+ "Current heap memory used by topic listings")
+ .create()
+ .setChild(new Gauge.Child() {
+ @Override
+ public double get() {
+ return heapMemoryLimiter.getAcquiredPermits();
+ }
+ }));
+ this.otelHeapMemoryUsedGauge = openTelemetryMeter.gaugeBuilder("topic.list.heap.memory.used")
+ .setUnit("By")
+ .setDescription("Current heap memory used by topic listings")
+ .buildWithCallback(observableDoubleMeasurement -> {
+ observableDoubleMeasurement.record(heapMemoryLimiter.getAcquiredPermits());
+ });
+
+ this.heapMemoryLimitBytes = register(Gauge.build(prometheusPrefix + "topic_list_heap_memory_limit_bytes",
+ "Configured heap memory limit")
+ .create());
+ this.heapMemoryLimitBytes.set(maxHeapMemory);
+ this.otelHeapMemoryLimitGauge = openTelemetryMeter.gaugeBuilder("topic.list.heap.memory.limit")
+ .setUnit("By")
+ .setDescription("Configured heap memory limit")
+ .build();
+ this.otelHeapMemoryLimitGauge.set(maxHeapMemory);
+
+ this.directMemoryUsedBytes = register(Gauge.build(prometheusPrefix + "topic_list_direct_memory_used_bytes",
+ "Current direct memory used by topic listings")
+ .create()
+ .setChild(new Gauge.Child() {
+ @Override
+ public double get() {
+ return directMemoryLimiter.getAcquiredPermits();
+ }
+ }));
+ this.otelDirectMemoryUsedGauge = openTelemetryMeter.gaugeBuilder("topic.list.direct.memory.used")
+ .setUnit("By")
+ .setDescription("Current direct memory used by topic listings")
+ .buildWithCallback(observableDoubleMeasurement -> {
+ observableDoubleMeasurement.record(directMemoryLimiter.getAcquiredPermits());
+ });
+
+ this.directMemoryLimitBytes = register(Gauge.build(prometheusPrefix + "topic_list_direct_memory_limit_bytes",
+ "Configured direct memory limit")
+ .create());
+ this.directMemoryLimitBytes.set(maxDirectMemory);
+ this.otelDirectMemoryLimitGauge = openTelemetryMeter.gaugeBuilder("topic.list.direct.memory.limit")
+ .setUnit("By")
+ .setDescription("Configured direct memory limit")
+ .build();
+ this.otelDirectMemoryLimitGauge.set(maxDirectMemory);
+
+ this.heapQueueSize = register(Gauge.build(prometheusPrefix + "topic_list_heap_queue_size",
+ "Current heap memory limiter queue size")
+ .create()
+ .setChild(new Gauge.Child() {
+ @Override
+ public double get() {
+ return heapMemoryLimiter.getQueueSize();
+ }
+ }));
+ this.otelHeapQueueSize = openTelemetryMeter
+ .upDownCounterBuilder("topic.list.heap.queue.size")
+ .setDescription("Current heap memory limiter queue size")
+ .setUnit("1")
+ .buildWithCallback(observableLongMeasurement -> {
+ observableLongMeasurement.record(heapMemoryLimiter.getQueueSize());
+ });
+
+ this.heapQueueMaxSize = register(Gauge.build(prometheusPrefix + "topic_list_heap_queue_max_size",
+ "Maximum heap memory limiter queue size")
+ .create());
+ this.heapQueueMaxSize.set(maxHeapQueueSize);
+ LongGauge otelHeapQueueMaxSize = openTelemetryMeter
+ .gaugeBuilder("topic.list.heap.queue.max.size")
+ .setDescription("Maximum heap memory limiter queue size")
+ .setUnit("1")
+ .ofLongs()
+ .build();
+ otelHeapQueueMaxSize.set(maxHeapQueueSize);
+
+ this.directQueueSize = register(Gauge.build(prometheusPrefix + "topic_list_direct_queue_size",
+ "Current direct memory limiter queue size")
+ .create()
+ .setChild(new Gauge.Child() {
+ @Override
+ public double get() {
+ return directMemoryLimiter.getQueueSize();
+ }
+ }));
+ this.otelDirectQueueSize = openTelemetryMeter
+ .upDownCounterBuilder("topic.list.direct.queue.size")
+ .setDescription("Current direct memory limiter queue size")
+ .setUnit("1")
+ .buildWithCallback(observableLongMeasurement -> {
+ observableLongMeasurement.record(directMemoryLimiter.getQueueSize());
+ });
+
+ this.directQueueMaxSize = register(Gauge.build(prometheusPrefix + "topic_list_direct_queue_max_size",
+ "Maximum direct memory limiter queue size")
+ .create());
+ this.directQueueMaxSize.set(maxDirectQueueSize);
+ LongGauge otelDirectQueueMaxSize = openTelemetryMeter
+ .gaugeBuilder("topic.list.direct.queue.max.size")
+ .setDescription("Maximum direct memory limiter queue size")
+ .setUnit("1")
+ .ofLongs()
+ .build();
+ otelDirectQueueMaxSize.set(maxDirectQueueSize);
+
+ this.heapWaitTimeMs = register(Summary.build(prometheusPrefix + "topic_list_heap_wait_time_ms",
+ "Wait time for heap memory permits")
+ .quantile(0.50, 0.01)
+ .quantile(0.95, 0.01)
+ .quantile(0.99, 0.01)
+ .quantile(1, 0.01)
+ .create());
+ this.otelHeapWaitTime = openTelemetryMeter.histogramBuilder("topic.list.heap.wait.time")
+ .setUnit("s")
+ .setDescription("Wait time for heap memory permits")
+ .build();
+
+ this.directWaitTimeMs = register(Summary.build(prometheusPrefix + "topic_list_direct_wait_time_ms",
Review Comment:
Same as the above comment
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/ServerCnx.java:
##########
@@ -2610,6 +2581,99 @@ protected void handleGetTopicsOfNamespace(CommandGetTopicsOfNamespace commandGet
}
}
+ private void internalHandleGetTopicsOfNamespace(String namespace, NamespaceName namespaceName, long requestId,
+ CommandGetTopicsOfNamespace.Mode mode,
+ Optional<String> topicsPattern, Optional<String> topicsHash,
+ Semaphore lookupSemaphore) {
+ BooleanSupplier isPermitRequestCancelled = () -> !ctx().channel().isActive();
+ TopicListSizeResultCache.ResultHolder
+ listSizeHolder = service.getTopicListSizeResultCache().getTopicListSize(namespaceName.toString(), mode);
+ listSizeHolder.getSizeAsync().thenAccept(initialSize -> {
+ maxTopicListInFlightLimiter.withAcquiredPermits(initialSize,
+ AsyncDualMemoryLimiter.LimitType.HEAP_MEMORY, isPermitRequestCancelled, initialPermits -> {
+ return getBrokerService().pulsar().getNamespaceService()
+ .getListOfUserTopics(namespaceName, mode)
+ .thenAccept(topics -> {
+ long actualSize = topics.stream()
+ .mapToInt(ByteBufUtil::utf8Bytes) // convert character count to bytes
+ .map(n -> n + 32) // add 32 bytes overhead for each entry
Review Comment:
Is there any reason to add extra 32 bytes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2440992635
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/ServerCnx.java:
##########
@@ -2610,6 +2581,99 @@ protected void handleGetTopicsOfNamespace(CommandGetTopicsOfNamespace commandGet
}
}
+ private void internalHandleGetTopicsOfNamespace(String namespace, NamespaceName namespaceName, long requestId,
+ CommandGetTopicsOfNamespace.Mode mode,
+ Optional<String> topicsPattern, Optional<String> topicsHash,
+ Semaphore lookupSemaphore) {
+ BooleanSupplier isPermitRequestCancelled = () -> !ctx().channel().isActive();
+ TopicListSizeResultCache.ResultHolder
+ listSizeHolder = service.getTopicListSizeResultCache().getTopicListSize(namespaceName.toString(), mode);
+ listSizeHolder.getSizeAsync().thenAccept(initialSize -> {
+ maxTopicListInFlightLimiter.withAcquiredPermits(initialSize,
+ AsyncDualMemoryLimiter.LimitType.HEAP_MEMORY, isPermitRequestCancelled, initialPermits -> {
+ return getBrokerService().pulsar().getNamespaceService()
+ .getListOfUserTopics(namespaceName, mode)
+ .thenAccept(topics -> {
+ long actualSize = topics.stream()
+ .mapToInt(ByteBufUtil::utf8Bytes) // convert character count to bytes
+ .map(n -> n + 32) // add 32 bytes overhead for each entry
Review Comment:
I have refactored the code and moved the calculation to a separate utility method in org.apache.pulsar.broker.topiclistlimit.TopicListMemoryLimiter#estimateTopicListSize . That contains a comment about the 16 byte object header and additional overhead of java.lang.String fields.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "poorbarcode (via GitHub)" <gi...@apache.org>.
poorbarcode commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2444987116
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/ServerCnx.java:
##########
@@ -2610,6 +2581,93 @@ protected void handleGetTopicsOfNamespace(CommandGetTopicsOfNamespace commandGet
}
}
+ private void internalHandleGetTopicsOfNamespace(String namespace, NamespaceName namespaceName, long requestId,
+ CommandGetTopicsOfNamespace.Mode mode,
+ Optional<String> topicsPattern, Optional<String> topicsHash,
+ Semaphore lookupSemaphore) {
+ BooleanSupplier isPermitRequestCancelled = () -> !ctx().channel().isActive();
+ TopicListSizeResultCache.ResultHolder
+ listSizeHolder = service.getTopicListSizeResultCache().getTopicListSize(namespaceName.toString(), mode);
+ listSizeHolder.getSizeAsync().thenAccept(initialSize -> {
+ maxTopicListInFlightLimiter.withAcquiredPermits(initialSize,
+ AsyncDualMemoryLimiter.LimitType.HEAP_MEMORY, isPermitRequestCancelled, initialPermits -> {
+ return getBrokerService().pulsar().getNamespaceService()
+ .getListOfUserTopics(namespaceName, mode)
+ .thenAccept(topics -> {
+ long actualSize = TopicListMemoryLimiter.estimateTopicListSize(topics);
+ listSizeHolder.updateSize(actualSize);
+ maxTopicListInFlightLimiter.withUpdatedPermits(initialPermits, actualSize,
Review Comment:
Instead of updating and acquiring additional permits if the actual size is larger than estimated, we'd better improve as follows:
- The current request does not need to acquire permits again; only updating the permits that were borrowed out is enough, even though the permits were over-acquired
- Updates the permits that were borrowed out, to limit the following permits acquisition;
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/ServerCnx.java:
##########
@@ -2610,6 +2581,93 @@ protected void handleGetTopicsOfNamespace(CommandGetTopicsOfNamespace commandGet
}
}
+ private void internalHandleGetTopicsOfNamespace(String namespace, NamespaceName namespaceName, long requestId,
+ CommandGetTopicsOfNamespace.Mode mode,
+ Optional<String> topicsPattern, Optional<String> topicsHash,
+ Semaphore lookupSemaphore) {
+ BooleanSupplier isPermitRequestCancelled = () -> !ctx().channel().isActive();
+ TopicListSizeResultCache.ResultHolder
+ listSizeHolder = service.getTopicListSizeResultCache().getTopicListSize(namespaceName.toString(), mode);
+ listSizeHolder.getSizeAsync().thenAccept(initialSize -> {
+ maxTopicListInFlightLimiter.withAcquiredPermits(initialSize,
+ AsyncDualMemoryLimiter.LimitType.HEAP_MEMORY, isPermitRequestCancelled, initialPermits -> {
Review Comment:
Just add a tip: if the data does not exist in the cache, it may query ZK, which will also use direct memory
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2466096541
##########
pulsar-broker-common/src/main/java/org/apache/pulsar/broker/topiclistlimit/TopicListSizeResultCache.java:
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.topiclistlimit;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReference;
+import org.apache.pulsar.common.api.proto.CommandGetTopicsOfNamespace;
+
+/**
+ * A cache for storing and managing topic list size estimates in namespaces.
+ * This class provides functionality to:
+ * - Cache and retrieve topic list size estimates for different namespaces and modes
+ * - Handle concurrent requests for topic list sizes efficiently
+ * - Maintain and update size estimates based on actual topic list sizes
+ * - Prevent thundering herd problems when multiple concurrent requests for a namespace are made without
+ * a previous size estimate
+ * The cache uses namespace name and topic list mode (PERSISTENT/NON_PERSISTENT/ALL) as keys
+ * and maintains size estimates that are refined with actual usage.
+ */
+public class TopicListSizeResultCache {
+ // 10kB initial estimate for topic list heap size
+ private static final long INITIAL_TOPIC_LIST_HEAP_SIZE = 10 * 1024;
+
+ private Cache<CacheKey, ResultHolder> topicListSizeCache = Caffeine.newBuilder()
+ .maximumSize(10000)
+ .build();
+
+ record CacheKey(String namespaceName, CommandGetTopicsOfNamespace.Mode mode) {
+ }
+
+ /**
+ * Holds the topic list size estimate and future for the topic list size.
+ * The size is returned by calling {@link #getSizeAsync()} method which is asynchronous.
+ * The estimate is updated by calling {@link #updateSize(long)} method.
+ */
+ public static class ResultHolder {
+ private final AtomicReference<CompletableFuture<Long>> topicListSizeFuture =
+ new AtomicReference<>(null);
+ private final AtomicLong existingSizeRef = new AtomicLong(-1L);
Review Comment:
It is needed. Please read the javadoc for getSizeAsync
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2466082661
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/TopicListService.java:
##########
@@ -101,6 +112,7 @@ public void accept(String topicName, NotificationType notificationType) {
private final boolean enableSubscriptionPatternEvaluation;
private final int maxSubscriptionPatternLength;
private final ConcurrentLongHashMap<CompletableFuture<TopicListWatcher>> watchers;
+ private final Backoff retryBackoff;
public TopicListService(PulsarService pulsar, ServerCnx connection,
Review Comment:
> This function is a notification for the client, to let the client know which topic was changed; it should not be limited by the new feature.
I disagree. To properly limit memory usage, it's also necessary to handle CommandWatchTopicListSuccess / CommandWatchTopicUpdate as explained in https://github.com/apache/pulsar/blob/master/pip/pip-442.md .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2429680066
##########
pulsar-broker-common/src/main/java/org/apache/pulsar/broker/topiclistlimit/TopicListMemoryLimiter.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.topiclistlimit;
+
+import io.opentelemetry.api.metrics.DoubleGauge;
+import io.opentelemetry.api.metrics.DoubleHistogram;
+import io.opentelemetry.api.metrics.LongCounter;
+import io.opentelemetry.api.metrics.LongGauge;
+import io.opentelemetry.api.metrics.Meter;
+import io.opentelemetry.api.metrics.ObservableDoubleGauge;
+import io.opentelemetry.api.metrics.ObservableLongUpDownCounter;
+import io.prometheus.client.Collector;
+import io.prometheus.client.CollectorRegistry;
+import io.prometheus.client.Counter;
+import io.prometheus.client.Gauge;
+import io.prometheus.client.Summary;
+import java.util.concurrent.TimeUnit;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.common.semaphore.AsyncDualMemoryLimiterImpl;
+import org.apache.pulsar.common.semaphore.AsyncSemaphore;
+
+/**
+ * Topic list memory limiter that exposes both Prometheus metrics and OpenTelemetry metrics.
+ */
+@Slf4j
+public class TopicListMemoryLimiter extends AsyncDualMemoryLimiterImpl {
+ private final CollectorRegistry collectorRegistry;
+ private final Gauge heapMemoryUsedBytes;
+ private final Gauge heapMemoryLimitBytes;
+ private final Gauge directMemoryUsedBytes;
+ private final Gauge directMemoryLimitBytes;
+ private final Gauge heapQueueSize;
+ private final Gauge heapQueueMaxSize;
+ private final Gauge directQueueSize;
+ private final Gauge directQueueMaxSize;
+ private final Summary heapWaitTimeMs;
+ private final Summary directWaitTimeMs;
+ private final Counter heapTimeoutTotal;
+ private final Counter directTimeoutTotal;
+ private final ObservableDoubleGauge otelHeapMemoryUsedGauge;
+ private final DoubleGauge otelHeapMemoryLimitGauge;
+ private final ObservableDoubleGauge otelDirectMemoryUsedGauge;
+ private final DoubleGauge otelDirectMemoryLimitGauge;
+ private final ObservableLongUpDownCounter otelHeapQueueSize;
+ private final ObservableLongUpDownCounter otelDirectQueueSize;
+ private final DoubleHistogram otelHeapWaitTime;
+ private final DoubleHistogram otelDirectWaitTime;
+ private final LongCounter otelHeapTimeoutTotal;
+ private final LongCounter otelDirectTimeoutTotal;
+
+ public TopicListMemoryLimiter(CollectorRegistry collectorRegistry, String prometheusPrefix,
+ Meter openTelemetryMeter,
+ long maxHeapMemory, int maxHeapQueueSize,
+ long heapTimeoutMillis, long maxDirectMemory, int maxDirectQueueSize,
+ long directTimeoutMillis) {
+ super(maxHeapMemory, maxHeapQueueSize, heapTimeoutMillis, maxDirectMemory, maxDirectQueueSize,
+ directTimeoutMillis);
+ this.collectorRegistry = collectorRegistry;
+
+ AsyncSemaphore heapMemoryLimiter = getLimiter(LimitType.HEAP_MEMORY);
+ AsyncSemaphore directMemoryLimiter = getLimiter(LimitType.DIRECT_MEMORY);
+
+ this.heapMemoryUsedBytes = register(Gauge.build(prometheusPrefix + "topic_list_heap_memory_used_bytes",
+ "Current heap memory used by topic listings")
+ .create()
+ .setChild(new Gauge.Child() {
+ @Override
+ public double get() {
+ return heapMemoryLimiter.getAcquiredPermits();
+ }
+ }));
+ this.otelHeapMemoryUsedGauge = openTelemetryMeter.gaugeBuilder("topic.list.heap.memory.used")
+ .setUnit("By")
+ .setDescription("Current heap memory used by topic listings")
+ .buildWithCallback(observableDoubleMeasurement -> {
+ observableDoubleMeasurement.record(heapMemoryLimiter.getAcquiredPermits());
+ });
+
+ this.heapMemoryLimitBytes = register(Gauge.build(prometheusPrefix + "topic_list_heap_memory_limit_bytes",
+ "Configured heap memory limit")
+ .create());
+ this.heapMemoryLimitBytes.set(maxHeapMemory);
+ this.otelHeapMemoryLimitGauge = openTelemetryMeter.gaugeBuilder("topic.list.heap.memory.limit")
+ .setUnit("By")
+ .setDescription("Configured heap memory limit")
+ .build();
+ this.otelHeapMemoryLimitGauge.set(maxHeapMemory);
+
+ this.directMemoryUsedBytes = register(Gauge.build(prometheusPrefix + "topic_list_direct_memory_used_bytes",
+ "Current direct memory used by topic listings")
+ .create()
+ .setChild(new Gauge.Child() {
+ @Override
+ public double get() {
+ return directMemoryLimiter.getAcquiredPermits();
+ }
+ }));
+ this.otelDirectMemoryUsedGauge = openTelemetryMeter.gaugeBuilder("topic.list.direct.memory.used")
+ .setUnit("By")
+ .setDescription("Current direct memory used by topic listings")
+ .buildWithCallback(observableDoubleMeasurement -> {
+ observableDoubleMeasurement.record(directMemoryLimiter.getAcquiredPermits());
+ });
+
+ this.directMemoryLimitBytes = register(Gauge.build(prometheusPrefix + "topic_list_direct_memory_limit_bytes",
+ "Configured direct memory limit")
+ .create());
+ this.directMemoryLimitBytes.set(maxDirectMemory);
+ this.otelDirectMemoryLimitGauge = openTelemetryMeter.gaugeBuilder("topic.list.direct.memory.limit")
+ .setUnit("By")
+ .setDescription("Configured direct memory limit")
+ .build();
+ this.otelDirectMemoryLimitGauge.set(maxDirectMemory);
+
+ this.heapQueueSize = register(Gauge.build(prometheusPrefix + "topic_list_heap_queue_size",
+ "Current heap memory limiter queue size")
+ .create()
+ .setChild(new Gauge.Child() {
+ @Override
+ public double get() {
+ return heapMemoryLimiter.getQueueSize();
+ }
+ }));
+ this.otelHeapQueueSize = openTelemetryMeter
+ .upDownCounterBuilder("topic.list.heap.queue.size")
+ .setDescription("Current heap memory limiter queue size")
+ .setUnit("1")
+ .buildWithCallback(observableLongMeasurement -> {
+ observableLongMeasurement.record(heapMemoryLimiter.getQueueSize());
+ });
+
+ this.heapQueueMaxSize = register(Gauge.build(prometheusPrefix + "topic_list_heap_queue_max_size",
+ "Maximum heap memory limiter queue size")
+ .create());
+ this.heapQueueMaxSize.set(maxHeapQueueSize);
+ LongGauge otelHeapQueueMaxSize = openTelemetryMeter
+ .gaugeBuilder("topic.list.heap.queue.max.size")
+ .setDescription("Maximum heap memory limiter queue size")
+ .setUnit("1")
+ .ofLongs()
+ .build();
+ otelHeapQueueMaxSize.set(maxHeapQueueSize);
+
+ this.directQueueSize = register(Gauge.build(prometheusPrefix + "topic_list_direct_queue_size",
+ "Current direct memory limiter queue size")
+ .create()
+ .setChild(new Gauge.Child() {
+ @Override
+ public double get() {
+ return directMemoryLimiter.getQueueSize();
+ }
+ }));
+ this.otelDirectQueueSize = openTelemetryMeter
+ .upDownCounterBuilder("topic.list.direct.queue.size")
+ .setDescription("Current direct memory limiter queue size")
+ .setUnit("1")
+ .buildWithCallback(observableLongMeasurement -> {
+ observableLongMeasurement.record(directMemoryLimiter.getQueueSize());
+ });
+
+ this.directQueueMaxSize = register(Gauge.build(prometheusPrefix + "topic_list_direct_queue_max_size",
+ "Maximum direct memory limiter queue size")
+ .create());
+ this.directQueueMaxSize.set(maxDirectQueueSize);
+ LongGauge otelDirectQueueMaxSize = openTelemetryMeter
+ .gaugeBuilder("topic.list.direct.queue.max.size")
+ .setDescription("Maximum direct memory limiter queue size")
+ .setUnit("1")
+ .ofLongs()
+ .build();
+ otelDirectQueueMaxSize.set(maxDirectQueueSize);
+
+ this.heapWaitTimeMs = register(Summary.build(prometheusPrefix + "topic_list_heap_wait_time_ms",
+ "Wait time for heap memory permits")
+ .quantile(0.50, 0.01)
+ .quantile(0.95, 0.01)
+ .quantile(0.99, 0.01)
+ .quantile(1, 0.01)
+ .create());
+ this.otelHeapWaitTime = openTelemetryMeter.histogramBuilder("topic.list.heap.wait.time.ms")
+ .setUnit("s")
+ .setDescription("Wait time for heap memory permits")
+ .build();
+
+ this.directWaitTimeMs = register(Summary.build(prometheusPrefix + "topic_list_direct_wait_time_ms",
+ "Wait time for direct memory permits")
+ .quantile(0.50, 0.01)
+ .quantile(0.95, 0.01)
+ .quantile(0.99, 0.01)
+ .quantile(1, 0.01)
+ .create());
+ this.otelDirectWaitTime = openTelemetryMeter.histogramBuilder("topic.list.direct.wait.time.ms")
+ .setUnit("s")
+ .setDescription("Wait time for direct memory permits")
+ .build();
+
+ this.heapTimeoutTotal = register(Counter.build(prometheusPrefix + "topic_list_heap_timeout_total",
+ "Total heap memory permit timeouts")
+ .create());
+ this.otelHeapTimeoutTotal = openTelemetryMeter.counterBuilder("topic.list.heap.timeout.total")
+ .setDescription("Total heap memory permit timeouts")
+ .setUnit("1")
+ .build();
+
+ this.directTimeoutTotal = register(Counter.build(prometheusPrefix + "topic_list_direct_timeout_total",
+ "Total direct memory permit timeouts")
+ .create());
+ this.otelDirectTimeoutTotal = openTelemetryMeter.counterBuilder("topic.list.direct.timeout.total")
+ .setDescription("Total direct memory permit timeouts")
+ .setUnit("1")
+ .build();
+ }
+
+ private <T extends Collector> T register(T collector) {
+ try {
+ collectorRegistry.register(collector);
+ } catch (Exception e) {
+ // ignore exception when registering a collector that is already registered
+ if (log.isDebugEnabled()) {
+ log.debug("Failed to register Prometheus collector {}", collector, e);
+ }
+ }
+ return collector;
+ }
+
+ private void unregister(Collector collector) {
+ try {
+ collectorRegistry.unregister(collector);
+ } catch (Exception e) {
+ // ignore exception when unregistering a collector that is not registered
+ if (log.isDebugEnabled()) {
+ log.debug("Failed to unregister Prometheus collector {}", collector, e);
+ }
+ }
+ }
+
+
+ @Override
+ protected void recordHeapWaitTime(long waitTimeNanos) {
+ if (waitTimeNanos == Long.MAX_VALUE) {
+ heapTimeoutTotal.inc();
+ otelHeapTimeoutTotal.add(1);
+ } else {
+ heapWaitTimeMs.observe(TimeUnit.NANOSECONDS.toMillis(waitTimeNanos));
+ otelHeapWaitTime.record(waitTimeNanos / 1_000_000_000.0d);
+ }
+ }
+
+ @Override
+ protected void recordDirectWaitTime(long waitTimeNanos) {
+ if (waitTimeNanos == Long.MAX_VALUE) {
+ directTimeoutTotal.inc();
+ otelDirectTimeoutTotal.add(1);
+ } else {
+ directWaitTimeMs.observe(TimeUnit.NANOSECONDS.toMillis(waitTimeNanos));
+ otelDirectWaitTime.record(waitTimeNanos / 1_000_000_000.0d);
+ }
+ }
+
+ @Override
+ public void close() {
+ super.close();
+ unregister(heapMemoryUsedBytes);
+ unregister(heapMemoryLimitBytes);
+ unregister(directMemoryUsedBytes);
+ unregister(directMemoryLimitBytes);
+ unregister(heapQueueSize);
+ unregister(heapQueueMaxSize);
+ unregister(directQueueSize);
+ unregister(directQueueMaxSize);
+ unregister(heapWaitTimeMs);
+ unregister(directWaitTimeMs);
+ unregister(heapTimeoutTotal);
+ unregister(directTimeoutTotal);
+ otelHeapMemoryUsedGauge.close();
+ otelDirectMemoryUsedGauge.close();
+ otelHeapQueueSize.close();
Review Comment:
there are no such methods on Otel instruments that aren't observable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2429777462
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/TopicListService.java:
##########
@@ -288,8 +294,13 @@ public void deleteTopicListWatcher(Long watcherId) {
*/
public void sendTopicListUpdate(long watcherId, String topicsHash, List<String> deletedTopics,
List<String> newTopics) {
- connection.getCommandSender().sendWatchTopicListUpdate(watcherId, newTopics, deletedTopics, topicsHash);
+ connection.getCommandSender().sendWatchTopicListUpdate(watcherId, newTopics, deletedTopics, topicsHash,
+ t -> {
+ // TODO add retry with backoff
+ log.warn(
+ "[{}] Cannot acquire direct memory tokens for sending topic list update. State will be "
+ + "inconsistent on the client. {}",
+ connection.toString(), t.getMessage());
+ });
Review Comment:
PTAL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2429841981
##########
pulsar-broker/src/test/java/org/apache/pulsar/client/api/PatternConsumerBackPressureMultipleConsumersTest.java:
##########
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.client.api;
+
+import static org.testng.Assert.assertEquals;
+import io.netty.buffer.ByteBuf;
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.UUID;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.Cleanup;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.auth.MockedPulsarServiceBaseTest;
+import org.apache.pulsar.broker.stats.prometheus.PrometheusMetricsClient;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.allocator.PulsarByteBufAllocator;
+import org.apache.pulsar.common.api.proto.CommandGetTopicsOfNamespace;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.stats.JvmMetrics;
+import org.apache.pulsar.common.util.DirectMemoryUtils;
+import org.testng.annotations.AfterMethod;
+import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.Test;
+
+@Slf4j
+@Test(groups = "broker-impl")
+public class PatternConsumerBackPressureMultipleConsumersTest extends MockedPulsarServiceBaseTest {
+
+ @Override
+ @BeforeMethod
+ protected void setup() throws Exception {
+ isTcpLookup = true;
+ super.internalSetup();
+ setupDefaultTenantAndNamespace();
+ }
+
+ @Override
+ @AfterMethod(alwaysRun = true)
+ protected void cleanup() throws Exception {
+ super.internalCleanup();
+ }
+
+ @Test(timeOut = 60 * 1000)
+ public void testGetTopicsWithLargeAmountOfConcurrentClientConnections()
+ throws PulsarAdminException, InterruptedException, IOException {
+ // number of requests to send to the broker
+ final int requests = 500;
+ // use multiple clients so that each client has a separate connection to the broker
+ final int numberOfClients = 200;
+ // create a long topic name to consume more memory per topic
+ final String topicName = StringUtils.repeat('a', 512) + UUID.randomUUID();
+ // number of topics to create
+ final int topicCount = 8192;
+ // maximum number of requests in flight at any given time
+ final int maxRequestsInFlight = 200;
+
+ // create a single topic with multiple partitions
+ admin.topics().createPartitionedTopic(topicName, topicCount);
+
+ // reduce available direct memory to reproduce issues with less concurrency
+ long directMemoryRequired = getDirectMemoryRequiredMB() * 1024 * 1024;
+ List<ByteBuf> buffers = allocateDirectMemory(directMemoryRequired);
+ @Cleanup
+ Closeable releaseBuffers = () -> {
+ for (ByteBuf byteBuf : buffers) {
+ byteBuf.release();
+ }
+ };
+
+ @Cleanup("shutdownNow")
+ final ExecutorService executorService = Executors.newFixedThreadPool(Runtime.getRuntime()
+ .availableProcessors());
+
+ @Cleanup
+ PulsarClientSharedResources sharedResources =
+ PulsarClientSharedResources.builder().build();
+ List<PulsarClientImpl> clients = new ArrayList<>(requests);
+ @Cleanup
+ Closeable closeClients = () -> {
+ for (PulsarClient client : clients) {
+ try {
+ client.close();
+ } catch (PulsarClientException e) {
+ log.error("Failed to close client {}", client, e);
+ }
+ }
+ };
+ for (int i = 0; i < numberOfClients; i++) {
+ PulsarClientImpl client = (PulsarClientImpl) PulsarClient.builder()
+ .serviceUrl(getClientServiceUrl())
+ .sharedResources(sharedResources)
+ .build();
+ clients.add(client);
+ }
+
+ final AtomicInteger success = new AtomicInteger(0);
+ final CountDownLatch latch = new CountDownLatch(requests);
+ final Semaphore semaphore = new Semaphore(maxRequestsInFlight);
+ for (int i = 0; i < requests; i++) {
+ PulsarClientImpl pulsarClientImpl = clients.get(i % numberOfClients);
+ executorService.execute(() -> {
+ semaphore.acquireUninterruptibly();
+ try {
+ pulsarClientImpl.getLookup()
+ .getTopicsUnderNamespace(NamespaceName.get("public", "default"),
+ CommandGetTopicsOfNamespace.Mode.PERSISTENT, ".*", "")
+ .whenComplete((result, ex) -> {
+ semaphore.release();
+ if (ex == null) {
+ success.incrementAndGet();
+ } else {
+ log.error("Failed to get topic list.", ex);
+ }
+ log.info("latch-count: {}, succeed: {}, available direct mem: {} MB", latch.getCount(),
+ success.get(),
+ (DirectMemoryUtils.jvmMaxDirectMemory() - JvmMetrics.getJvmDirectMemoryUsed())
+ / (1024 * 1024));
+ latch.countDown();
+ });
+ } catch (Exception e) {
+ semaphore.release();
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2466087403
##########
pulsar-broker-common/src/main/java/org/apache/pulsar/broker/topiclistlimit/TopicListMemoryLimiter.java:
##########
@@ -0,0 +1,312 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.topiclistlimit;
+
+import io.netty.buffer.ByteBufUtil;
+import io.opentelemetry.api.metrics.DoubleGauge;
+import io.opentelemetry.api.metrics.DoubleHistogram;
+import io.opentelemetry.api.metrics.LongCounter;
+import io.opentelemetry.api.metrics.LongGauge;
+import io.opentelemetry.api.metrics.Meter;
+import io.opentelemetry.api.metrics.ObservableDoubleGauge;
+import io.opentelemetry.api.metrics.ObservableLongUpDownCounter;
+import io.prometheus.client.Collector;
+import io.prometheus.client.CollectorRegistry;
+import io.prometheus.client.Counter;
+import io.prometheus.client.Gauge;
+import io.prometheus.client.Summary;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.common.semaphore.AsyncDualMemoryLimiterImpl;
+import org.apache.pulsar.common.semaphore.AsyncSemaphore;
+
+/**
+ * Topic list memory limiter that exposes both Prometheus metrics and OpenTelemetry metrics.
+ */
+@Slf4j
+public class TopicListMemoryLimiter extends AsyncDualMemoryLimiterImpl {
Review Comment:
> Do we need to indicate in this class name that this is a class related to metrics?
I don't think so. There are many classes in Pulsar that contain metrics and it's not part of the class name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2503808541
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/TopicListService.java:
##########
@@ -101,6 +112,7 @@ public void accept(String topicName, NotificationType notificationType) {
private final boolean enableSubscriptionPatternEvaluation;
private final int maxSubscriptionPatternLength;
private final ConcurrentLongHashMap<CompletableFuture<TopicListWatcher>> watchers;
+ private final Backoff retryBackoff;
public TopicListService(PulsarService pulsar, ServerCnx connection,
Review Comment:
@poorbarcode It appears that previous changes have broke the topic list watcher functionality. In the current Pulsar client code, the CommandWatchTopicListSuccess response is ignored:
https://github.com/apache/pulsar/blame/1300f0a4b1c015e278ff321d58c12a7e85ba12e7/pulsar-client/src/main/java/org/apache/pulsar/client/impl/TopicListWatcher.java#L136-L152
This was changed in #21853 while fixing the lost topic list events issue. While working on the code, I found the actual root cause that causes topic list events to be lost. I'll fix that in a separate PR.
What I'll do now is that I'll split the Topic list watcher memory limiting to another PR from this current PR. That is useful since the PR to fix the topic list events issue is not related to PIP-442. After that is fixed, it makes sense to continue adding PIP-442 to cover also topic list events.
This way we can also make progress on getting this current PR merged, without the topic list watcher limiting changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2490511409
##########
pulsar-common/src/main/java/org/apache/pulsar/common/semaphore/AsyncSemaphoreImpl.java:
##########
@@ -0,0 +1,330 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.common.semaphore;
+
+import io.netty.util.concurrent.DefaultThreadFactory;
+import java.util.Queue;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicLongFieldUpdater;
+import java.util.function.BooleanSupplier;
+import java.util.function.LongConsumer;
+import org.apache.pulsar.common.util.Runnables;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of AsyncSemaphore with timeout and queue size limits.
+ */
+public class AsyncSemaphoreImpl implements AsyncSemaphore, AutoCloseable {
+ private static final Logger log = LoggerFactory.getLogger(AsyncSemaphoreImpl.class);
+
+ private final AtomicLong availablePermits;
+ private final Queue<PendingRequest> queue;
+ private final long maxPermits;
+ private final long timeoutMillis;
+ private final ScheduledExecutorService executor;
+ private final boolean shutdownExecutor;
+ private final LongConsumer queueLatencyRecorder;
+ private final AtomicBoolean closed = new AtomicBoolean(false);
+ private final Runnable processQueueRunnable = Runnables.catchingAndLoggingThrowables(this::internalProcessQueue);
+
+ /**
+ * Creates an AsyncSemaphoreImpl with the given parameters.
+ * @param maxPermits max number of permits available for acquisition, set to <= 0 for unbounded semaphore (not
+ * recommended)
+ * @param maxQueueSize max number of requests that can be queued, set to <= 0 for unbounded queue (not recommended)
+ * @param timeoutMillis timeout in milliseconds for acquiring permits
+ */
+ public AsyncSemaphoreImpl(long maxPermits, int maxQueueSize, long timeoutMillis) {
+ this(maxPermits, maxQueueSize, timeoutMillis, maxPermits > 0 ? createExecutor() : null, maxPermits > 0, null);
+ }
+
+ /**
+ * Creates an AsyncSemaphoreImpl with the given parameters.
+ * @param maxPermits max number of permits available for acquisition, set to <= 0 for unbounded semaphore (not
+ * recommended)
+ * @param maxQueueSize max number of requests that can be queued, set to <= 0 for unbounded queue (not recommended)
+ * @param timeoutMillis timeout in milliseconds for acquiring permits
+ * @param executor executor service to use for scheduling timeouts, it is expected to be single threaded
+ * @param queueLatencyRecorder consumer to record queue latency, Long.MAX_VALUE is used for requests that timed out
+ */
+ public AsyncSemaphoreImpl(long maxPermits, int maxQueueSize, long timeoutMillis,
+ ScheduledExecutorService executor, LongConsumer queueLatencyRecorder) {
+ this(maxPermits, maxQueueSize, timeoutMillis, executor, false, queueLatencyRecorder);
+ }
+
+ AsyncSemaphoreImpl(long maxPermits, int maxQueueSize, long timeoutMillis, ScheduledExecutorService executor,
+ boolean shutdownExecutor, LongConsumer queueLatencyRecorder) {
+ this.availablePermits = new AtomicLong(maxPermits);
+ this.maxPermits = maxPermits;
+ this.queue = maxQueueSize > 0 ? new ArrayBlockingQueue<>(maxQueueSize) : new LinkedBlockingQueue<>();
+ this.timeoutMillis = timeoutMillis;
+ this.executor = executor;
+ this.shutdownExecutor = shutdownExecutor;
+ this.queueLatencyRecorder = queueLatencyRecorder;
+ }
+
+ private static ScheduledExecutorService createExecutor() {
+ return Executors.newSingleThreadScheduledExecutor(
+ new DefaultThreadFactory("async-semaphore-executor"));
+ }
+
+ @Override
+ public CompletableFuture<AsyncSemaphorePermit> acquire(long permits, BooleanSupplier isCancelled) {
+ return internalAcquire(permits, permits, isCancelled);
+ }
+
+ private CompletableFuture<AsyncSemaphorePermit> internalAcquire(long permits, long acquirePermits,
+ BooleanSupplier isCancelled) {
+ validatePermits(permits);
+
+ // if maximum permits is <= 0, then the semaphore is unbounded
+ if (isUnbounded()) {
+ return CompletableFuture.completedFuture(new SemaphorePermit(permits));
+ }
+
+ CompletableFuture<AsyncSemaphorePermit> future = new CompletableFuture<>();
+
+ if (closed.get()) {
+ future.completeExceptionally(new PermitAcquireAlreadyClosedException("Semaphore is closed"));
+ return future;
+ }
+
+ PendingRequest request = new PendingRequest(permits, acquirePermits, future, isCancelled);
+ if (!queue.offer(request)) {
+ future.completeExceptionally(new PermitAcquireQueueFullException(
+ "Semaphore queue is full"));
+ return future;
+ }
+ // Schedule timeout
+ ScheduledFuture<?> timeoutTask = executor.schedule(() -> {
+ if (!request.future.isDone() && queue.remove(request)) {
+ // timeout is recorded with Long.MAX_VALUE as the age
+ recordQueueLatency(Long.MAX_VALUE);
+ // also record the time in the queue
+ recordQueueLatency(request.getAgeNanos());
+ future.completeExceptionally(new PermitAcquireTimeoutException(
+ "Permit acquisition timed out"));
+ // the next request might have smaller permits and that might be processed
+ processQueue();
+ }
+ }, timeoutMillis, TimeUnit.MILLISECONDS);
+ request.setTimeoutTask(timeoutTask);
+
+ processQueue();
+ return future;
+ }
+
+ private void validatePermits(long permits) {
+ if (permits < 0) {
+ throw new IllegalArgumentException("Invalid negative permits value: " + permits);
+ }
+ if (!isUnbounded() && permits > maxPermits) {
+ throw new IllegalArgumentException(
+ "Requested permits=" + permits + " is larger than maxPermits=" + maxPermits);
+ }
+ }
+
+ private boolean isUnbounded() {
+ return maxPermits <= 0;
+ }
+
+ private void recordQueueLatency(long ageNanos) {
+ if (queueLatencyRecorder != null) {
+ queueLatencyRecorder.accept(ageNanos);
+ }
+ }
+
+ @Override
+ public CompletableFuture<AsyncSemaphorePermit> update(AsyncSemaphorePermit permit, long newPermits,
+ BooleanSupplier isCancelled) {
+ validatePermits(newPermits);
+ if (isUnbounded()) {
+ return CompletableFuture.completedFuture(new SemaphorePermit(newPermits));
+ }
+ long oldPermits = permit.getPermits();
+ long additionalPermits = newPermits - oldPermits;
+ // mark the old permits as released without adding the permits to availablePermits
+ castToImplementation(permit).releasePermits();
+ if (additionalPermits > 0) {
+ return internalAcquire(newPermits, additionalPermits, isCancelled);
+ } else {
+ // new permits are less than the old ones, so we return the difference
+ availablePermits.addAndGet(-additionalPermits);
+ processQueue();
+ // return the new permits immediately
+ return CompletableFuture.completedFuture(new SemaphorePermit(newPermits));
Review Comment:
Tests added in 578f027
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#issuecomment-3386969884
## [Codecov](https://app.codecov.io/gh/apache/pulsar/pull/24833?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `81.11455%` with `61 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 74.28%. Comparing base ([`4457b08`](https://app.codecov.io/gh/apache/pulsar/commit/4457b089daf815d009f89c88c8e4bc35a8233d54?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`aad9a6f`](https://app.codecov.io/gh/apache/pulsar/commit/aad9a6f3d9828c26158586ea7a5fccba70db924c?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/pulsar/pull/24833?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...va/org/apache/pulsar/broker/service/ServerCnx.java](https://app.codecov.io/gh/apache/pulsar/pull/24833?src=pr&el=tree&filepath=pulsar-broker%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fbroker%2Fservice%2FServerCnx.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9zZXJ2aWNlL1NlcnZlckNueC5qYXZh) | 54.90% | [21 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/pulsar/pull/24833?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...apache/pulsar/proxy/server/LookupProxyHandler.java](https://app.codecov.io/gh/apache/pulsar/pull/24833?src=pr&el=tree&filepath=pulsar-proxy%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fproxy%2Fserver%2FLookupProxyHandler.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLXByb3h5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9wdWxzYXIvcHJveHkvc2VydmVyL0xvb2t1cFByb3h5SGFuZGxlci5qYXZh) | 42.85% | [18 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/pulsar/pull/24833?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...he/pulsar/common/semaphore/AsyncSemaphoreImpl.java](https://app.codecov.io/gh/apache/pulsar/pull/24833?src=pr&el=tree&filepath=pulsar-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fcommon%2Fsemaphore%2FAsyncSemaphoreImpl.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLWNvbW1vbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NvbW1vbi9zZW1hcGhvcmUvQXN5bmNTZW1hcGhvcmVJbXBsLmphdmE=) | 93.39% | [3 Missing and 4 partials :warning: ](https://app.codecov.io/gh/apache/pulsar/pull/24833?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...apache/pulsar/broker/service/TopicListService.java](https://app.codecov.io/gh/apache/pulsar/pull/24833?src=pr&el=tree&filepath=pulsar-broker%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fbroker%2Fservice%2FTopicListService.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9zZXJ2aWNlL1RvcGljTGlzdFNlcnZpY2UuamF2YQ==) | 25.00% | [6 Missing :warning: ](https://app.codecov.io/gh/apache/pulsar/pull/24833?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...pulsar/broker/service/PulsarCommandSenderImpl.java](https://app.codecov.io/gh/apache/pulsar/pull/24833?src=pr&el=tree&filepath=pulsar-broker%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fbroker%2Fservice%2FPulsarCommandSenderImpl.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9zZXJ2aWNlL1B1bHNhckNvbW1hbmRTZW5kZXJJbXBsLmphdmE=) | 50.00% | [0 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/pulsar/pull/24833?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...r/common/semaphore/AsyncDualMemoryLimiterImpl.java](https://app.codecov.io/gh/apache/pulsar/pull/24833?src=pr&el=tree&filepath=pulsar-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fcommon%2Fsemaphore%2FAsyncDualMemoryLimiterImpl.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLWNvbW1vbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NvbW1vbi9zZW1hcGhvcmUvQXN5bmNEdWFsTWVtb3J5TGltaXRlckltcGwuamF2YQ==) | 95.12% | [1 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/pulsar/pull/24833?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
[](https://app.codecov.io/gh/apache/pulsar/pull/24833?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
```diff
@@ Coverage Diff @@
## master #24833 +/- ##
=============================================
+ Coverage 38.39% 74.28% +35.88%
- Complexity 13123 33866 +20743
=============================================
Files 1856 1918 +62
Lines 145015 149445 +4430
Branches 16818 17316 +498
=============================================
+ Hits 55684 111012 +55328
+ Misses 81787 29599 -52188
- Partials 7544 8834 +1290
```
| [Flag](https://app.codecov.io/gh/apache/pulsar/pull/24833/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [inttests](https://app.codecov.io/gh/apache/pulsar/pull/24833/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `26.74% <17.64%> (+0.43%)` | :arrow_up: |
| [systests](https://app.codecov.io/gh/apache/pulsar/pull/24833/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `22.85% <51.08%> (+0.09%)` | :arrow_up: |
| [unittests](https://app.codecov.io/gh/apache/pulsar/pull/24833/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `73.81% <81.11%> (+39.27%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/pulsar/pull/24833?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...org/apache/pulsar/broker/ServiceConfiguration.java](https://app.codecov.io/gh/apache/pulsar/pull/24833?src=pr&el=tree&filepath=pulsar-broker-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fbroker%2FServiceConfiguration.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLWJyb2tlci1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3B1bHNhci9icm9rZXIvU2VydmljZUNvbmZpZ3VyYXRpb24uamF2YQ==) | `98.22% <100.00%> (+3.11%)` | :arrow_up: |
| [...rg/apache/pulsar/broker/service/BrokerService.java](https://app.codecov.io/gh/apache/pulsar/pull/24833?src=pr&el=tree&filepath=pulsar-broker%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fbroker%2Fservice%2FBrokerService.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9zZXJ2aWNlL0Jyb2tlclNlcnZpY2UuamF2YQ==) | `83.59% <100.00%> (+24.81%)` | :arrow_up: |
| [...va/org/apache/pulsar/common/protocol/Commands.java](https://app.codecov.io/gh/apache/pulsar/pull/24833?src=pr&el=tree&filepath=pulsar-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fcommon%2Fprotocol%2FCommands.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLWNvbW1vbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NvbW1vbi9wcm90b2NvbC9Db21tYW5kcy5qYXZh) | `90.99% <100.00%> (+24.87%)` | :arrow_up: |
| [...ulsar/common/semaphore/AsyncDualMemoryLimiter.java](https://app.codecov.io/gh/apache/pulsar/pull/24833?src=pr&el=tree&filepath=pulsar-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fcommon%2Fsemaphore%2FAsyncDualMemoryLimiter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLWNvbW1vbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NvbW1vbi9zZW1hcGhvcmUvQXN5bmNEdWFsTWVtb3J5TGltaXRlci5qYXZh) | `100.00% <100.00%> (ø)` | |
| [...r/common/semaphore/AsyncDualMemoryLimiterUtil.java](https://app.codecov.io/gh/apache/pulsar/pull/24833?src=pr&el=tree&filepath=pulsar-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fcommon%2Fsemaphore%2FAsyncDualMemoryLimiterUtil.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLWNvbW1vbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NvbW1vbi9zZW1hcGhvcmUvQXN5bmNEdWFsTWVtb3J5TGltaXRlclV0aWwuamF2YQ==) | `100.00% <100.00%> (ø)` | |
| [...apache/pulsar/common/semaphore/AsyncSemaphore.java](https://app.codecov.io/gh/apache/pulsar/pull/24833?src=pr&el=tree&filepath=pulsar-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fcommon%2Fsemaphore%2FAsyncSemaphore.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLWNvbW1vbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NvbW1vbi9zZW1hcGhvcmUvQXN5bmNTZW1hcGhvcmUuamF2YQ==) | `100.00% <100.00%> (ø)` | |
| [...apache/pulsar/proxy/server/ProxyConfiguration.java](https://app.codecov.io/gh/apache/pulsar/pull/24833?src=pr&el=tree&filepath=pulsar-proxy%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fproxy%2Fserver%2FProxyConfiguration.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLXByb3h5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9wdWxzYXIvcHJveHkvc2VydmVyL1Byb3h5Q29uZmlndXJhdGlvbi5qYXZh) | `95.67% <100.00%> (+16.19%)` | :arrow_up: |
| [...a/org/apache/pulsar/proxy/server/ProxyService.java](https://app.codecov.io/gh/apache/pulsar/pull/24833?src=pr&el=tree&filepath=pulsar-proxy%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fproxy%2Fserver%2FProxyService.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLXByb3h5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9wdWxzYXIvcHJveHkvc2VydmVyL1Byb3h5U2VydmljZS5qYXZh) | `80.75% <100.00%> (+31.72%)` | :arrow_up: |
| [...r/common/semaphore/AsyncDualMemoryLimiterImpl.java](https://app.codecov.io/gh/apache/pulsar/pull/24833?src=pr&el=tree&filepath=pulsar-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fcommon%2Fsemaphore%2FAsyncDualMemoryLimiterImpl.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLWNvbW1vbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NvbW1vbi9zZW1hcGhvcmUvQXN5bmNEdWFsTWVtb3J5TGltaXRlckltcGwuamF2YQ==) | `95.12% <95.12%> (ø)` | |
| [...pulsar/broker/service/PulsarCommandSenderImpl.java](https://app.codecov.io/gh/apache/pulsar/pull/24833?src=pr&el=tree&filepath=pulsar-broker%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fbroker%2Fservice%2FPulsarCommandSenderImpl.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9zZXJ2aWNlL1B1bHNhckNvbW1hbmRTZW5kZXJJbXBsLmphdmE=) | `85.93% <50.00%> (+24.08%)` | :arrow_up: |
| ... and [4 more](https://app.codecov.io/gh/apache/pulsar/pull/24833?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | |
... and [1409 files with indirect coverage changes](https://app.codecov.io/gh/apache/pulsar/pull/24833/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "Denovo1998 (via GitHub)" <gi...@apache.org>.
Denovo1998 commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2426451067
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/ServerCnx.java:
##########
@@ -2606,6 +2577,77 @@ protected void handleGetTopicsOfNamespace(CommandGetTopicsOfNamespace commandGet
}
}
+ private void internalHandleGetTopicsOfNamespace(String namespace, NamespaceName namespaceName, long requestId,
+ CommandGetTopicsOfNamespace.Mode mode,
+ Optional<String> topicsPattern, Optional<String> topicsHash,
+ Semaphore lookupSemaphore) {
+ BooleanSupplier isPermitRequestCancelled = () -> !ctx().channel().isActive();
+ maxTopicListInFlightLimiter.withAcquiredPermits(INITIAL_TOPIC_LIST_HEAP_PERMITS_SIZE,
+ AsyncDualMemoryLimiter.LimitType.HEAP_MEMORY, isPermitRequestCancelled, initialPermits -> {
+ return getBrokerService().pulsar().getNamespaceService().getListOfUserTopics(namespaceName, mode)
+ .thenAccept(topics -> {
+ long actualSize = topics.stream().mapToInt(String::length).sum();
+ maxTopicListInFlightLimiter.withUpdatedPermits(initialPermits, actualSize,
+ isPermitRequestCancelled, permits -> {
+ boolean filterTopics = false;
+ // filter system topic
+ List<String> filteredTopics = topics;
+
+ if (enableSubscriptionPatternEvaluation && topicsPattern.isPresent()) {
+ if (topicsPattern.get().length() <= maxSubscriptionPatternLength) {
+ filterTopics = true;
+ filteredTopics = TopicList.filterTopics(filteredTopics, topicsPattern.get(),
+ topicsPatternImplementation);
+ } else {
+ log.info("[{}] Subscription pattern provided [{}] was longer than "
+ + "maximum {}.", remoteAddress, topicsPattern.get(),
+ maxSubscriptionPatternLength);
+ }
+ }
+ String hash = TopicList.calculateHash(filteredTopics);
+ boolean hashUnchanged = topicsHash.isPresent() && topicsHash.get().equals(hash);
+ if (hashUnchanged) {
+ filteredTopics = Collections.emptyList();
+ }
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Received CommandGetTopicsOfNamespace for namespace "
+ + "[//{}] by {}, size:{}", remoteAddress, namespace,
+ requestId,
+ topics.size());
+ }
+ commandSender.sendGetTopicsOfNamespaceResponse(filteredTopics, hash, filterTopics,
+ !hashUnchanged, requestId, ex -> {
+ log.warn("[{}] Failed to acquire direct memory permits for "
+ + "GetTopicsOfNamespace: {}", remoteAddress, ex.getMessage());
+ commandSender.sendErrorResponse(requestId, ServerError.TooManyRequests,
+ "Cannot acquire permits for direct memory");
+ });
+ return CompletableFuture.completedFuture(null);
+ }, t -> {
+ log.warn("[{}] Failed to acquire heap memory permits for "
+ + "GetTopicsOfNamespace: {}", remoteAddress, t.getMessage());
+ writeAndFlush(Commands.newError(requestId, ServerError.TooManyRequests,
+ "Failed due to heap memory limit exceeded"));
+ return CompletableFuture.completedFuture(null);
+ });
+ }).whenComplete((__, ___) -> {
+ lookupSemaphore.release();
+ }).exceptionally(ex -> {
+ log.warn("[{}] Error GetTopicsOfNamespace for namespace [//{}] by {}", remoteAddress,
+ namespace, requestId);
+ commandSender.sendErrorResponse(requestId,
+ BrokerServiceException.getClientErrorCode(new ServerMetadataException(ex)),
+ ex.getMessage());
+ return null;
+ });
+ }, t -> {
+ log.warn("[{}] Failed to acquire initial heap memory permits for GetTopicsOfNamespace: {}",
+ remoteAddress, t.getMessage());
+ writeAndFlush(Commands.newError(requestId, ServerError.TooManyRequests,
+ "Failed due to heap memory limit exceeded"));
+ return CompletableFuture.completedFuture(null);
Review Comment:
Should we call `lookupSemaphore.release();` here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "Denovo1998 (via GitHub)" <gi...@apache.org>.
Denovo1998 commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2426420341
##########
org/apache/pulsar/common/util/memory/AsyncSemaphore.java:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.common.util.memory;
+
+import java.util.concurrent.CompletableFuture;
+
+/**
+ * Interface for asynchronous semaphore operations.
+ * Used for managing concurrent access to limited resources.
+ */
+public interface AsyncSemaphore {
Review Comment:
Are the five files in this directory redundant?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2426609426
##########
pulsar-common/src/main/java/org/apache/pulsar/common/semaphore/AsyncSemaphore.java:
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.common.semaphore;
+
+import java.util.concurrent.CompletableFuture;
+import java.util.function.BooleanSupplier;
+
+/**
+ * An abstraction for a generic asynchronous semaphore.
+ */
+public interface AsyncSemaphore {
+ /**
+ * Acquire permits from the semaphore.
+ * Returned future completes when permits are available.
+ * It will complete exceptionally with AsyncSemaphorePermitAcquireTimeoutException on timeout
+ * and exceptionally with AsyncSemaphorePermitAcquireQueueFullException when queue full
+ * @return CompletableFuture that completes with permit when available
+ */
+ CompletableFuture<AsyncSemaphorePermit> acquire(long permits, BooleanSupplier isCancelled);
+
+ /**
+ * Acquire or release permits for previously acquired permits by updating the permits.
+ * Returns a future that completes when permits are available.
+ * It will complete exceptionally with AsyncSemaphorePermitAcquireTimeoutException on timeout
+ * and exceptionally with AsyncSemaphorePermitAcquireQueueFullException when queue full
+ * @return CompletableFuture that completes with permit when available
+ */
Review Comment:
The PIP document has been updated to match this implementation. What's missing?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#issuecomment-3452301010
@codelipenghui @poorbarcode I've addressed the review comments. PTAL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2506853281
##########
pulsar-common/src/main/java/org/apache/pulsar/common/semaphore/AsyncDualMemoryLimiter.java:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.common.semaphore;
+
+import java.util.concurrent.CompletableFuture;
+import java.util.function.BooleanSupplier;
+import java.util.function.Function;
+
+/**
+ * An abstraction for an asynchronous memory semaphore that tracks separate limits for heap and direct memory.
+ */
+public interface AsyncDualMemoryLimiter {
+ enum LimitType {
+ HEAP_MEMORY, // For heap memory allocation
+ DIRECT_MEMORY // For direct memory allocation
+ }
+
+ /**
+ * Acquire permits for the specified memory size.
+ * Returned future completes when memory permits are available.
+ * It will complete exceptionally with AsyncSemaphore.PermitAcquireTimeoutException on timeout
+ * and exceptionally with AsyncSemaphore.PermitAcquireQueueFullException when queue full
+ *
+ * @param memorySize the size of memory to acquire permits for
+ * @param limitType the type of memory limit (HEAP_MEMORY or DIRECT_MEMORY)
+ * @param isCancelled supplier that returns true if acquisition should be cancelled
+ * @return CompletableFuture that completes with permit when available
+ */
+ CompletableFuture<AsyncDualMemoryLimiterPermit> acquire(long memorySize, LimitType limitType,
+ BooleanSupplier isCancelled);
+
+ /**
+ * Acquire or release permits for previously acquired permits by updating the requested memory size.
+ * Returns a future that completes when permits are available.
+ * It will complete exceptionally with AsyncSemaphore.PermitAcquireTimeoutException on timeout
+ * and exceptionally with AsyncSemaphore.PermitAcquireQueueFullException when queue full
+ * The provided permit is released when the permits are successfully acquired and the returned updated
+ * permit replaces the old instance.
+ *
+ * @param permit the previously acquired permit to update
+ * @param newMemorySize the new memory size to update to
+ * @param isCancelled supplier that returns true if update should be cancelled
+ * @return CompletableFuture that completes with permit when available
+ */
+ CompletableFuture<AsyncDualMemoryLimiterPermit> update(AsyncDualMemoryLimiterPermit permit, long newMemorySize,
+ BooleanSupplier isCancelled);
+
+ /**
+ * Release previously acquired permit.
+ * Must be called to prevent memory permit leaks.
+ *
+ * @param permit the permit to release
+ */
+ void release(AsyncDualMemoryLimiterPermit permit);
+ /**
+ * Execute the specified function with acquired permits and release the permits after the returned future completes.
+ * @param memorySize memory size to acquire permits for
+ * @param limitType memory limit type to acquire permits for
+ * @param function function to execute with acquired permits
+ * @return result of the function
+ * @param <T> type of the CompletableFuture returned by the function
+ */
+ default <T> CompletableFuture<T> withAcquiredPermits(long memorySize, LimitType limitType,
+ BooleanSupplier isCancelled,
+ Function<AsyncDualMemoryLimiterPermit,
+ CompletableFuture<T>> function,
+ Function<Throwable, CompletableFuture<T>>
+ permitAcquireErrorHandler) {
+ return AsyncDualMemoryLimiterUtil.withPermitsFuture(acquire(memorySize, limitType, isCancelled), function,
+ permitAcquireErrorHandler, this::release);
+ }
+
+ /**
+ * Executed the specified function with updated permits and release the permits after the returned future completes.
Review Comment:
I'll fix in follow up PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "poorbarcode (via GitHub)" <gi...@apache.org>.
poorbarcode commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2491202010
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/TopicListService.java:
##########
@@ -101,6 +112,7 @@ public void accept(String topicName, NotificationType notificationType) {
private final boolean enableSubscriptionPatternEvaluation;
private final int maxSubscriptionPatternLength;
private final ConcurrentLongHashMap<CompletableFuture<TopicListWatcher>> watchers;
+ private final Backoff retryBackoff;
public TopicListService(PulsarService pulsar, ServerCnx connection,
Review Comment:
@lhotari The command should not be limited, because `MultiTopicsConsumer` and `MultiTopicsProducer` need to receive this command once the topics' partition is changed. The regexp consumer is also relies on the command.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "poorbarcode (via GitHub)" <gi...@apache.org>.
poorbarcode commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2487231319
##########
pulsar-common/src/main/java/org/apache/pulsar/common/semaphore/AsyncSemaphoreImpl.java:
##########
@@ -0,0 +1,330 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.common.semaphore;
+
+import io.netty.util.concurrent.DefaultThreadFactory;
+import java.util.Queue;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicLongFieldUpdater;
+import java.util.function.BooleanSupplier;
+import java.util.function.LongConsumer;
+import org.apache.pulsar.common.util.Runnables;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of AsyncSemaphore with timeout and queue size limits.
+ */
+public class AsyncSemaphoreImpl implements AsyncSemaphore, AutoCloseable {
+ private static final Logger log = LoggerFactory.getLogger(AsyncSemaphoreImpl.class);
+
+ private final AtomicLong availablePermits;
+ private final Queue<PendingRequest> queue;
+ private final long maxPermits;
+ private final long timeoutMillis;
+ private final ScheduledExecutorService executor;
+ private final boolean shutdownExecutor;
+ private final LongConsumer queueLatencyRecorder;
+ private final AtomicBoolean closed = new AtomicBoolean(false);
+ private final Runnable processQueueRunnable = Runnables.catchingAndLoggingThrowables(this::internalProcessQueue);
+
+ /**
+ * Creates an AsyncSemaphoreImpl with the given parameters.
+ * @param maxPermits max number of permits available for acquisition, set to <= 0 for unbounded semaphore (not
+ * recommended)
+ * @param maxQueueSize max number of requests that can be queued, set to <= 0 for unbounded queue (not recommended)
+ * @param timeoutMillis timeout in milliseconds for acquiring permits
+ */
+ public AsyncSemaphoreImpl(long maxPermits, int maxQueueSize, long timeoutMillis) {
+ this(maxPermits, maxQueueSize, timeoutMillis, maxPermits > 0 ? createExecutor() : null, maxPermits > 0, null);
+ }
+
+ /**
+ * Creates an AsyncSemaphoreImpl with the given parameters.
+ * @param maxPermits max number of permits available for acquisition, set to <= 0 for unbounded semaphore (not
+ * recommended)
+ * @param maxQueueSize max number of requests that can be queued, set to <= 0 for unbounded queue (not recommended)
+ * @param timeoutMillis timeout in milliseconds for acquiring permits
+ * @param executor executor service to use for scheduling timeouts, it is expected to be single threaded
+ * @param queueLatencyRecorder consumer to record queue latency, Long.MAX_VALUE is used for requests that timed out
+ */
+ public AsyncSemaphoreImpl(long maxPermits, int maxQueueSize, long timeoutMillis,
+ ScheduledExecutorService executor, LongConsumer queueLatencyRecorder) {
+ this(maxPermits, maxQueueSize, timeoutMillis, executor, false, queueLatencyRecorder);
+ }
+
+ AsyncSemaphoreImpl(long maxPermits, int maxQueueSize, long timeoutMillis, ScheduledExecutorService executor,
+ boolean shutdownExecutor, LongConsumer queueLatencyRecorder) {
+ this.availablePermits = new AtomicLong(maxPermits);
+ this.maxPermits = maxPermits;
+ this.queue = maxQueueSize > 0 ? new ArrayBlockingQueue<>(maxQueueSize) : new LinkedBlockingQueue<>();
+ this.timeoutMillis = timeoutMillis;
+ this.executor = executor;
+ this.shutdownExecutor = shutdownExecutor;
+ this.queueLatencyRecorder = queueLatencyRecorder;
+ }
+
+ private static ScheduledExecutorService createExecutor() {
+ return Executors.newSingleThreadScheduledExecutor(
+ new DefaultThreadFactory("async-semaphore-executor"));
+ }
+
+ @Override
+ public CompletableFuture<AsyncSemaphorePermit> acquire(long permits, BooleanSupplier isCancelled) {
+ return internalAcquire(permits, permits, isCancelled);
+ }
+
+ private CompletableFuture<AsyncSemaphorePermit> internalAcquire(long permits, long acquirePermits,
+ BooleanSupplier isCancelled) {
+ validatePermits(permits);
+
+ // if maximum permits is <= 0, then the semaphore is unbounded
+ if (isUnbounded()) {
+ return CompletableFuture.completedFuture(new SemaphorePermit(permits));
+ }
+
+ CompletableFuture<AsyncSemaphorePermit> future = new CompletableFuture<>();
+
+ if (closed.get()) {
+ future.completeExceptionally(new PermitAcquireAlreadyClosedException("Semaphore is closed"));
+ return future;
+ }
+
+ PendingRequest request = new PendingRequest(permits, acquirePermits, future, isCancelled);
+ if (!queue.offer(request)) {
+ future.completeExceptionally(new PermitAcquireQueueFullException(
+ "Semaphore queue is full"));
+ return future;
+ }
+ // Schedule timeout
+ ScheduledFuture<?> timeoutTask = executor.schedule(() -> {
+ if (!request.future.isDone() && queue.remove(request)) {
+ // timeout is recorded with Long.MAX_VALUE as the age
+ recordQueueLatency(Long.MAX_VALUE);
+ // also record the time in the queue
+ recordQueueLatency(request.getAgeNanos());
+ future.completeExceptionally(new PermitAcquireTimeoutException(
+ "Permit acquisition timed out"));
+ // the next request might have smaller permits and that might be processed
+ processQueue();
+ }
+ }, timeoutMillis, TimeUnit.MILLISECONDS);
+ request.setTimeoutTask(timeoutTask);
+
+ processQueue();
+ return future;
+ }
+
+ private void validatePermits(long permits) {
+ if (permits < 0) {
+ throw new IllegalArgumentException("Invalid negative permits value: " + permits);
+ }
+ if (!isUnbounded() && permits > maxPermits) {
+ throw new IllegalArgumentException(
+ "Requested permits=" + permits + " is larger than maxPermits=" + maxPermits);
+ }
+ }
+
+ private boolean isUnbounded() {
+ return maxPermits <= 0;
+ }
+
+ private void recordQueueLatency(long ageNanos) {
+ if (queueLatencyRecorder != null) {
+ queueLatencyRecorder.accept(ageNanos);
+ }
+ }
+
+ @Override
+ public CompletableFuture<AsyncSemaphorePermit> update(AsyncSemaphorePermit permit, long newPermits,
+ BooleanSupplier isCancelled) {
+ validatePermits(newPermits);
+ if (isUnbounded()) {
+ return CompletableFuture.completedFuture(new SemaphorePermit(newPermits));
+ }
+ long oldPermits = permit.getPermits();
+ long additionalPermits = newPermits - oldPermits;
+ // mark the old permits as released without adding the permits to availablePermits
+ castToImplementation(permit).releasePermits();
+ if (additionalPermits > 0) {
+ return internalAcquire(newPermits, additionalPermits, isCancelled);
+ } else {
+ // new permits are less than the old ones, so we return the difference
+ availablePermits.addAndGet(-additionalPermits);
+ processQueue();
+ // return the new permits immediately
+ return CompletableFuture.completedFuture(new SemaphorePermit(newPermits));
Review Comment:
You are right(there is no bug), I added a new test as follows
```java
public void testWithAcquiredAndUpdatedPermitsDoesntLeakPermitsWhenUpdatedSizeIsLarger() throws Exception {
assertThat(limiter.getLimiter(LimitType.HEAP_MEMORY).getAvailablePermits()).isEqualTo(10000);
CompletableFuture<Void> result = limiter.withAcquiredPermits(
90, // estimated permits
LimitType.HEAP_MEMORY,
() -> false,
firstPermit -> {
// Simulate getting actual size and updating permits
return limiter.withUpdatedPermits(
firstPermit,
100,
() -> false,
secondPermit -> {
System.out.println(0/0);
return CompletableFuture.completedFuture(null);
},
throwable -> CompletableFuture.failedFuture(throwable)
);
},
throwable -> CompletableFuture.failedFuture(throwable)
);
try {
result.get();
} catch (Exception e) {
// it is expected.
}
assertThat(limiter.getLimiter(LimitType.HEAP_MEMORY).getAcquiredPermits()).isEqualTo(0);
assertThat(limiter.getLimiter(LimitType.HEAP_MEMORY).getAvailablePermits()).isEqualTo(10000);
}
```
----
Sorry, I truly understand the entire process now.
- Util method: acquire permits
- Creates the first object AsyncDualMemoryLimiterPermit (`p1`)
- Util method: acquire permits
- Creates the second object AsyncDualMemoryLimiterPermit (`p2`)
- Sets the value of `p1` to `0`
- Releases permits
- Release `p2`, which maintains the full permits
- Release `p1`, it has been set to `0`
And another concept is also important:
- `SemaphorePermit.releasePermits` releases nothing, it just updates the permits it acquired
- `AsyncDualMemoryLimiter.release(permits)` does release the permits
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2429776383
##########
pulsar-broker-common/src/main/java/org/apache/pulsar/broker/topiclistlimit/TopicListMemoryLimiter.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.topiclistlimit;
+
+import io.opentelemetry.api.metrics.DoubleGauge;
+import io.opentelemetry.api.metrics.DoubleHistogram;
+import io.opentelemetry.api.metrics.LongCounter;
+import io.opentelemetry.api.metrics.LongGauge;
+import io.opentelemetry.api.metrics.Meter;
+import io.opentelemetry.api.metrics.ObservableDoubleGauge;
+import io.opentelemetry.api.metrics.ObservableLongUpDownCounter;
+import io.prometheus.client.Collector;
+import io.prometheus.client.CollectorRegistry;
+import io.prometheus.client.Counter;
+import io.prometheus.client.Gauge;
+import io.prometheus.client.Summary;
+import java.util.concurrent.TimeUnit;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.common.semaphore.AsyncDualMemoryLimiterImpl;
+import org.apache.pulsar.common.semaphore.AsyncSemaphore;
+
+/**
+ * Topic list memory limiter that exposes both Prometheus metrics and OpenTelemetry metrics.
+ */
+@Slf4j
+public class TopicListMemoryLimiter extends AsyncDualMemoryLimiterImpl {
+ private final CollectorRegistry collectorRegistry;
+ private final Gauge heapMemoryUsedBytes;
+ private final Gauge heapMemoryLimitBytes;
+ private final Gauge directMemoryUsedBytes;
+ private final Gauge directMemoryLimitBytes;
+ private final Gauge heapQueueSize;
+ private final Gauge heapQueueMaxSize;
+ private final Gauge directQueueSize;
+ private final Gauge directQueueMaxSize;
+ private final Summary heapWaitTimeMs;
+ private final Summary directWaitTimeMs;
+ private final Counter heapTimeoutTotal;
+ private final Counter directTimeoutTotal;
+ private final ObservableDoubleGauge otelHeapMemoryUsedGauge;
+ private final DoubleGauge otelHeapMemoryLimitGauge;
+ private final ObservableDoubleGauge otelDirectMemoryUsedGauge;
+ private final DoubleGauge otelDirectMemoryLimitGauge;
+ private final ObservableLongUpDownCounter otelHeapQueueSize;
+ private final ObservableLongUpDownCounter otelDirectQueueSize;
+ private final DoubleHistogram otelHeapWaitTime;
+ private final DoubleHistogram otelDirectWaitTime;
+ private final LongCounter otelHeapTimeoutTotal;
+ private final LongCounter otelDirectTimeoutTotal;
+
+ public TopicListMemoryLimiter(CollectorRegistry collectorRegistry, String prometheusPrefix,
+ Meter openTelemetryMeter,
+ long maxHeapMemory, int maxHeapQueueSize,
+ long heapTimeoutMillis, long maxDirectMemory, int maxDirectQueueSize,
+ long directTimeoutMillis) {
+ super(maxHeapMemory, maxHeapQueueSize, heapTimeoutMillis, maxDirectMemory, maxDirectQueueSize,
+ directTimeoutMillis);
+ this.collectorRegistry = collectorRegistry;
+
+ AsyncSemaphore heapMemoryLimiter = getLimiter(LimitType.HEAP_MEMORY);
+ AsyncSemaphore directMemoryLimiter = getLimiter(LimitType.DIRECT_MEMORY);
+
+ this.heapMemoryUsedBytes = register(Gauge.build(prometheusPrefix + "topic_list_heap_memory_used_bytes",
+ "Current heap memory used by topic listings")
+ .create()
+ .setChild(new Gauge.Child() {
+ @Override
+ public double get() {
+ return heapMemoryLimiter.getAcquiredPermits();
+ }
+ }));
+ this.otelHeapMemoryUsedGauge = openTelemetryMeter.gaugeBuilder("topic.list.heap.memory.used")
+ .setUnit("By")
+ .setDescription("Current heap memory used by topic listings")
+ .buildWithCallback(observableDoubleMeasurement -> {
+ observableDoubleMeasurement.record(heapMemoryLimiter.getAcquiredPermits());
+ });
+
+ this.heapMemoryLimitBytes = register(Gauge.build(prometheusPrefix + "topic_list_heap_memory_limit_bytes",
+ "Configured heap memory limit")
+ .create());
+ this.heapMemoryLimitBytes.set(maxHeapMemory);
+ this.otelHeapMemoryLimitGauge = openTelemetryMeter.gaugeBuilder("topic.list.heap.memory.limit")
+ .setUnit("By")
+ .setDescription("Configured heap memory limit")
+ .build();
+ this.otelHeapMemoryLimitGauge.set(maxHeapMemory);
+
+ this.directMemoryUsedBytes = register(Gauge.build(prometheusPrefix + "topic_list_direct_memory_used_bytes",
+ "Current direct memory used by topic listings")
+ .create()
+ .setChild(new Gauge.Child() {
+ @Override
+ public double get() {
+ return directMemoryLimiter.getAcquiredPermits();
+ }
+ }));
+ this.otelDirectMemoryUsedGauge = openTelemetryMeter.gaugeBuilder("topic.list.direct.memory.used")
+ .setUnit("By")
+ .setDescription("Current direct memory used by topic listings")
+ .buildWithCallback(observableDoubleMeasurement -> {
+ observableDoubleMeasurement.record(directMemoryLimiter.getAcquiredPermits());
+ });
+
+ this.directMemoryLimitBytes = register(Gauge.build(prometheusPrefix + "topic_list_direct_memory_limit_bytes",
+ "Configured direct memory limit")
+ .create());
+ this.directMemoryLimitBytes.set(maxDirectMemory);
+ this.otelDirectMemoryLimitGauge = openTelemetryMeter.gaugeBuilder("topic.list.direct.memory.limit")
+ .setUnit("By")
+ .setDescription("Configured direct memory limit")
+ .build();
+ this.otelDirectMemoryLimitGauge.set(maxDirectMemory);
+
+ this.heapQueueSize = register(Gauge.build(prometheusPrefix + "topic_list_heap_queue_size",
+ "Current heap memory limiter queue size")
+ .create()
+ .setChild(new Gauge.Child() {
+ @Override
+ public double get() {
+ return heapMemoryLimiter.getQueueSize();
+ }
+ }));
+ this.otelHeapQueueSize = openTelemetryMeter
+ .upDownCounterBuilder("topic.list.heap.queue.size")
+ .setDescription("Current heap memory limiter queue size")
+ .setUnit("1")
+ .buildWithCallback(observableLongMeasurement -> {
+ observableLongMeasurement.record(heapMemoryLimiter.getQueueSize());
+ });
+
+ this.heapQueueMaxSize = register(Gauge.build(prometheusPrefix + "topic_list_heap_queue_max_size",
+ "Maximum heap memory limiter queue size")
+ .create());
+ this.heapQueueMaxSize.set(maxHeapQueueSize);
+ LongGauge otelHeapQueueMaxSize = openTelemetryMeter
+ .gaugeBuilder("topic.list.heap.queue.max.size")
+ .setDescription("Maximum heap memory limiter queue size")
+ .setUnit("1")
+ .ofLongs()
+ .build();
+ otelHeapQueueMaxSize.set(maxHeapQueueSize);
+
+ this.directQueueSize = register(Gauge.build(prometheusPrefix + "topic_list_direct_queue_size",
+ "Current direct memory limiter queue size")
+ .create()
+ .setChild(new Gauge.Child() {
+ @Override
+ public double get() {
+ return directMemoryLimiter.getQueueSize();
+ }
+ }));
+ this.otelDirectQueueSize = openTelemetryMeter
+ .upDownCounterBuilder("topic.list.direct.queue.size")
+ .setDescription("Current direct memory limiter queue size")
+ .setUnit("1")
+ .buildWithCallback(observableLongMeasurement -> {
+ observableLongMeasurement.record(directMemoryLimiter.getQueueSize());
+ });
+
+ this.directQueueMaxSize = register(Gauge.build(prometheusPrefix + "topic_list_direct_queue_max_size",
+ "Maximum direct memory limiter queue size")
+ .create());
+ this.directQueueMaxSize.set(maxDirectQueueSize);
+ LongGauge otelDirectQueueMaxSize = openTelemetryMeter
+ .gaugeBuilder("topic.list.direct.queue.max.size")
+ .setDescription("Maximum direct memory limiter queue size")
+ .setUnit("1")
+ .ofLongs()
+ .build();
+ otelDirectQueueMaxSize.set(maxDirectQueueSize);
Review Comment:
there is no close method in Otel for instruments that aren't "observable".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2432238309
##########
pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/LookupProxyHandler.java:
##########
@@ -357,6 +358,62 @@ private void performGetTopicsOfNamespace(long clientRequestId,
});
}
+ private void internalPerformGetTopicsOfNamespace(long clientRequestId, String namespaceName, ClientCnx clientCnx,
+ ByteBuf command, long requestId) {
+ BooleanSupplier isPermitRequestCancelled = () -> !proxyConnection.ctx().channel().isActive();
+ maxTopicListInFlightLimiter.withAcquiredPermits(INITIAL_TOPIC_LIST_HEAP_PERMITS_SIZE,
+ AsyncDualMemoryLimiter.LimitType.HEAP_MEMORY, isPermitRequestCancelled, initialPermits -> {
+ return clientCnx.newGetTopicsOfNamespace(command, requestId).whenComplete((r, t) -> {
+ if (t != null) {
+ log.warn("[{}] Failed to get TopicsOfNamespace {}: {}", clientAddress, namespaceName,
+ t.getMessage());
+ writeAndFlush(Commands.newError(clientRequestId, getServerError(t), t.getMessage()));
+ } else {
+ long actualSize =
+ r.getNonPartitionedOrPartitionTopics().stream()
+ .mapToInt(ByteBufUtil::utf8Bytes) // convert character count to bytes
+ .map(n -> n + 32) // add 32 bytes overhead for each entry
Review Comment:
Sure, there's multiple ways to use the Streams API. I don't see why that would be any better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2429738228
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/TopicListService.java:
##########
@@ -288,8 +294,13 @@ public void deleteTopicListWatcher(Long watcherId) {
*/
public void sendTopicListUpdate(long watcherId, String topicsHash, List<String> deletedTopics,
List<String> newTopics) {
- connection.getCommandSender().sendWatchTopicListUpdate(watcherId, newTopics, deletedTopics, topicsHash);
+ connection.getCommandSender().sendWatchTopicListUpdate(watcherId, newTopics, deletedTopics, topicsHash,
+ t -> {
+ // TODO add retry with backoff
+ log.warn(
+ "[{}] Cannot acquire direct memory tokens for sending topic list update. State will be "
+ + "inconsistent on the client. {}",
+ connection.toString(), t.getMessage());
+ });
Review Comment:
I'm handling this with changes to address the retry with backoff which is currently a TODO. I'll push the changes very soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2421874203
##########
pulsar-common/src/main/java/org/apache/pulsar/common/semaphore/AsyncDualMemoryLimiterUtil.java:
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.common.semaphore;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.channel.ChannelHandlerContext;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.BooleanSupplier;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import lombok.experimental.UtilityClass;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.common.api.proto.BaseCommand;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.semaphore.AsyncDualMemoryLimiter.AsyncDualMemoryLimiterPermit;
+
+@UtilityClass
+public class AsyncDualMemoryLimiterUtil {
+
+ public static <T> CompletableFuture<T> withPermitsFuture(
+ CompletableFuture<AsyncDualMemoryLimiterPermit>
+ permitsFuture,
+ Function<AsyncDualMemoryLimiterPermit,
+ CompletableFuture<T>> function,
+ Function<Throwable, CompletableFuture<T>>
+ permitAcquireErrorHandler,
+ Consumer<AsyncDualMemoryLimiterPermit> releaser) {
+ return permitsFuture
+ // combine the permits and error into a single pair so that it can be used in thenCompose
+ .handle((permits, permitAcquireError) ->
+ Pair.of(permits, permitAcquireError))
+ .thenCompose(permitsAndError -> {
+ if (permitsAndError.getRight() != null) {
+ // permits weren't acquired
+ return permitAcquireErrorHandler.apply(permitsAndError.getRight());
+ } else {
+ // permits were acquired
+ AsyncDualMemoryLimiterPermit permits = permitsAndError.getLeft();
+ try {
+ return function.apply(permits)
+ .whenComplete((__, ___) ->
+ // release the permits
+ releaser.accept(permits));
+ } catch (Throwable t) {
+ // release the permits if an exception occurs before the function returns
+ releaser.accept(permits);
+ throw t;
+ }
+ }
+ });
+ }
+
+ /**
+ * Acquire permits and write the command as the response to the channel.
+ * Releases the permits after the response has been written to the socket or the write fails.
+ *
+ * @param ctx the channel handler context.
+ * @param dualMemoryLimiter the memory limiter to acquire permits from.
+ * @param command the command to write to the channel.
+ * @return a future that completes when the command has been written to the channel's outbound buffer.
+ */
+ public static CompletableFuture<Void> acquireDirectMemoryPermitsAndWriteAndFlush(ChannelHandlerContext ctx,
+ AsyncDualMemoryLimiter
+ dualMemoryLimiter,
+ BooleanSupplier isCancelled,
+ BaseCommand command,
+ Consumer<Throwable>
+ permitAcquireErrorHandler
+ ) {
+ // Calculate serialized size before acquiring permits
+ int serializedSize = command.getSerializedSize();
+ // Acquire permits
+ return dualMemoryLimiter.acquire(serializedSize, AsyncDualMemoryLimiter.LimitType.DIRECT_MEMORY, isCancelled)
+ .whenComplete((permits, t) -> {
+ if (t != null) {
+ permitAcquireErrorHandler.accept(t);
+ return;
+ }
+ try {
+ // Serialize the response
+ ByteBuf outBuf = Commands.serializeWithPrecalculatedSerializedSize(command, serializedSize);
+ // Write the response
+ ctx.writeAndFlush(outBuf).addListener(future -> {
+ // Release permits after the response has been written to the socket
+ dualMemoryLimiter.release(permits);
+ });
+ } catch (Exception e) {
+ // Return permits if an exception occurs before writeAndFlush is called successfully
+ dualMemoryLimiter.release(permits);
+ throw e;
+ }
+ }).thenAccept(__ -> {
Review Comment:
The purpose is to map to CompletableFuture<Void>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "poorbarcode (via GitHub)" <gi...@apache.org>.
poorbarcode commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2443607042
##########
pulsar-broker-common/src/main/java/org/apache/pulsar/broker/topiclistlimit/TopicListSizeResultCache.java:
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.topiclistlimit;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReference;
+import org.apache.pulsar.common.api.proto.CommandGetTopicsOfNamespace;
+
+/**
+ * A cache for storing and managing topic list size estimates in namespaces.
+ * This class provides functionality to:
+ * - Cache and retrieve topic list size estimates for different namespaces and modes
+ * - Handle concurrent requests for topic list sizes efficiently
+ * - Maintain and update size estimates based on actual topic list sizes
+ * - Prevent thundering herd problems when multiple concurrent requests for a namespace are made without
+ * a previous size estimate
+ * The cache uses namespace name and topic list mode (PERSISTENT/NON_PERSISTENT/ALL) as keys
+ * and maintains size estimates that are refined with actual usage.
+ */
+public class TopicListSizeResultCache {
+ // 10kB initial estimate for topic list heap size
+ private static final long INITIAL_TOPIC_LIST_HEAP_SIZE = 10 * 1024;
+
+ private Cache<CacheKey, ResultHolder> topicListSizeCache = Caffeine.newBuilder()
+ .maximumSize(10000)
+ .build();
+
+ record CacheKey(String namespaceName, CommandGetTopicsOfNamespace.Mode mode) {
+ }
+
+ /**
+ * Holds the topic list size estimate and future for the topic list size.
+ * The size is returned by calling {@link #getSizeAsync()} method which is asynchronous.
+ * The estimate is updated by calling {@link #updateSize(long)} method.
+ */
+ public static class ResultHolder {
+ private final AtomicReference<CompletableFuture<Long>> topicListSizeFuture =
Review Comment:
The value of the topic list size seems never to be modified asynchronously, so the type being CompletableFuture is unnecessary.
##########
pulsar-broker-common/src/main/java/org/apache/pulsar/broker/topiclistlimit/TopicListSizeResultCache.java:
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.topiclistlimit;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReference;
+import org.apache.pulsar.common.api.proto.CommandGetTopicsOfNamespace;
+
+/**
+ * A cache for storing and managing topic list size estimates in namespaces.
+ * This class provides functionality to:
+ * - Cache and retrieve topic list size estimates for different namespaces and modes
+ * - Handle concurrent requests for topic list sizes efficiently
+ * - Maintain and update size estimates based on actual topic list sizes
+ * - Prevent thundering herd problems when multiple concurrent requests for a namespace are made without
+ * a previous size estimate
+ * The cache uses namespace name and topic list mode (PERSISTENT/NON_PERSISTENT/ALL) as keys
+ * and maintains size estimates that are refined with actual usage.
+ */
+public class TopicListSizeResultCache {
+ // 10kB initial estimate for topic list heap size
+ private static final long INITIAL_TOPIC_LIST_HEAP_SIZE = 10 * 1024;
+
+ private Cache<CacheKey, ResultHolder> topicListSizeCache = Caffeine.newBuilder()
+ .maximumSize(10000)
+ .build();
+
+ record CacheKey(String namespaceName, CommandGetTopicsOfNamespace.Mode mode) {
+ }
+
+ /**
+ * Holds the topic list size estimate and future for the topic list size.
+ * The size is returned by calling {@link #getSizeAsync()} method which is asynchronous.
+ * The estimate is updated by calling {@link #updateSize(long)} method.
+ */
+ public static class ResultHolder {
+ private final AtomicReference<CompletableFuture<Long>> topicListSizeFuture =
+ new AtomicReference<>(null);
+ private final AtomicLong existingSizeRef = new AtomicLong(-1L);
Review Comment:
Since the value of the topic list size seems never to be modified asynchronously, the unnecessary field `topicListSizeFuture` can be removed.
##########
pulsar-broker-common/src/main/java/org/apache/pulsar/broker/topiclistlimit/TopicListSizeResultCache.java:
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.topiclistlimit;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReference;
+import org.apache.pulsar.common.api.proto.CommandGetTopicsOfNamespace;
+
+/**
+ * A cache for storing and managing topic list size estimates in namespaces.
+ * This class provides functionality to:
+ * - Cache and retrieve topic list size estimates for different namespaces and modes
+ * - Handle concurrent requests for topic list sizes efficiently
+ * - Maintain and update size estimates based on actual topic list sizes
+ * - Prevent thundering herd problems when multiple concurrent requests for a namespace are made without
+ * a previous size estimate
+ * The cache uses namespace name and topic list mode (PERSISTENT/NON_PERSISTENT/ALL) as keys
+ * and maintains size estimates that are refined with actual usage.
+ */
+public class TopicListSizeResultCache {
+ // 10kB initial estimate for topic list heap size
+ private static final long INITIAL_TOPIC_LIST_HEAP_SIZE = 10 * 1024;
+
+ private Cache<CacheKey, ResultHolder> topicListSizeCache = Caffeine.newBuilder()
+ .maximumSize(10000)
+ .build();
+
+ record CacheKey(String namespaceName, CommandGetTopicsOfNamespace.Mode mode) {
+ }
+
+ /**
+ * Holds the topic list size estimate and future for the topic list size.
+ * The size is returned by calling {@link #getSizeAsync()} method which is asynchronous.
+ * The estimate is updated by calling {@link #updateSize(long)} method.
+ */
+ public static class ResultHolder {
+ private final AtomicReference<CompletableFuture<Long>> topicListSizeFuture =
+ new AtomicReference<>(null);
+ private final AtomicLong existingSizeRef = new AtomicLong(-1L);
+
+ /**
+ * Get the topic list size estimate. The first request will return the initial estimate
+ * and update the estimate based on the returned size of the topic list. Other concurrent requests
+ * will wait for the first request to complete and use the estimate of the first request.
+ * Subsequent requests will use estimate which gets updated based on the returned size of the topic list
+ * of each request.
+ * @return a future that will return the topic list size estimate
+ */
+ public CompletableFuture<Long> getSizeAsync() {
+ if (topicListSizeFuture.compareAndSet(null, new CompletableFuture<>())) {
+ // let the first request proceed with the initial estimate
+ return CompletableFuture.completedFuture(INITIAL_TOPIC_LIST_HEAP_SIZE);
+ } else {
+ // all other requests wait for the first one to complete
+ return topicListSizeFuture.get();
+ }
+ }
+
+ /**
+ * Update the topic list size estimate. The new estimated size will be updated by calculating the average
+ * of the existing and the new size. If the difference between the new and the existing size is less than 1,
+ * no update will be done.
+ * @param actualSize the actual size of the topic list
+ */
+ public void updateSize(long actualSize) {
Review Comment:
We need to mention that the current modifying operation may be dropped at line 103
```java
if (currentFuture != null && !currentFuture.isDone()) {
currentFuture.complete(existingSizeValue);
} else if (currentFuture == null || currentFuture.join().longValue() != existingSizeValue) {
topicListSizeFuture.compareAndSet(currentFuture, CompletableFuture.completedFuture(existingSizeValue));
} else {
// The operation may be dropped here
}
```
##########
pulsar-common/src/main/java/org/apache/pulsar/common/semaphore/AsyncSemaphoreImpl.java:
##########
@@ -0,0 +1,328 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.common.semaphore;
+
+import io.netty.util.concurrent.DefaultThreadFactory;
+import java.util.Queue;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicLongFieldUpdater;
+import java.util.function.BooleanSupplier;
+import java.util.function.LongConsumer;
+import org.apache.pulsar.common.util.Runnables;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of AsyncSemaphore with timeout and queue size limits.
+ */
+public class AsyncSemaphoreImpl implements AsyncSemaphore, AutoCloseable {
+ private static final Logger log = LoggerFactory.getLogger(AsyncSemaphoreImpl.class);
+
+ private final AtomicLong availablePermits;
+ private final Queue<PendingRequest> queue;
+ private final long maxPermits;
+ private final long timeoutMillis;
+ private final ScheduledExecutorService executor;
+ private final boolean shutdownExecutor;
+ private final LongConsumer queueLatencyRecorder;
+ private final AtomicBoolean closed = new AtomicBoolean(false);
+ private final Runnable processQueueRunnable = Runnables.catchingAndLoggingThrowables(this::internalProcessQueue);
+
+ /**
+ * Creates an AsyncSemaphoreImpl with the given parameters.
+ * @param maxPermits max number of permits available for acquisition, set to <= 0 for unbounded semaphore (not
+ * recommended)
+ * @param maxQueueSize max number of requests that can be queued, set to <= 0 for unbounded queue (not recommended)
+ * @param timeoutMillis timeout in milliseconds for acquiring permits
+ */
+ public AsyncSemaphoreImpl(long maxPermits, int maxQueueSize, long timeoutMillis) {
+ this(maxPermits, maxQueueSize, timeoutMillis, maxPermits > 0 ? createExecutor() : null, maxPermits > 0, null);
+ }
+
+ /**
+ * Creates an AsyncSemaphoreImpl with the given parameters.
+ * @param maxPermits max number of permits available for acquisition, set to <= 0 for unbounded semaphore (not
+ * recommended)
+ * @param maxQueueSize max number of requests that can be queued, set to <= 0 for unbounded queue (not recommended)
+ * @param timeoutMillis timeout in milliseconds for acquiring permits
+ * @param executor executor service to use for scheduling timeouts, it is expected to be single threaded
+ * @param queueLatencyRecorder consumer to record queue latency, Long.MAX_VALUE is used for requests that timed out
+ */
+ public AsyncSemaphoreImpl(long maxPermits, int maxQueueSize, long timeoutMillis,
+ ScheduledExecutorService executor, LongConsumer queueLatencyRecorder) {
+ this(maxPermits, maxQueueSize, timeoutMillis, executor, false, queueLatencyRecorder);
+ }
+
+ AsyncSemaphoreImpl(long maxPermits, int maxQueueSize, long timeoutMillis, ScheduledExecutorService executor,
+ boolean shutdownExecutor, LongConsumer queueLatencyRecorder) {
+ this.availablePermits = new AtomicLong(maxPermits);
+ this.maxPermits = maxPermits;
+ this.queue = maxQueueSize > 0 ? new ArrayBlockingQueue<>(maxQueueSize) : new LinkedBlockingQueue<>();
+ this.timeoutMillis = timeoutMillis;
+ this.executor = executor;
+ this.shutdownExecutor = shutdownExecutor;
+ this.queueLatencyRecorder = queueLatencyRecorder;
+ }
+
+ private static ScheduledExecutorService createExecutor() {
+ return Executors.newSingleThreadScheduledExecutor(
+ new DefaultThreadFactory("async-semaphore-executor"));
+ }
+
+ @Override
+ public CompletableFuture<AsyncSemaphorePermit> acquire(long permits, BooleanSupplier isCancelled) {
+ return internalAcquire(permits, permits, isCancelled);
+ }
+
+ private CompletableFuture<AsyncSemaphorePermit> internalAcquire(long permits, long acquirePermits,
+ BooleanSupplier isCancelled) {
+ if (!isPermitsValidForAcquiring(permits)) {
Review Comment:
We'd better separate the following two cases:
- `permits <0`: it may be caused by a bug
- `permits > maxPermits`: users should reset the configuration
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "poorbarcode (via GitHub)" <gi...@apache.org>.
poorbarcode commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2492539745
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/TopicListService.java:
##########
@@ -101,6 +112,7 @@ public void accept(String topicName, NotificationType notificationType) {
private final boolean enableSubscriptionPatternEvaluation;
private final int maxSubscriptionPatternLength;
private final ConcurrentLongHashMap<CompletableFuture<TopicListWatcher>> watchers;
+ private final Backoff retryBackoff;
public TopicListService(PulsarService pulsar, ServerCnx connection,
Review Comment:
> unless it times out or the queue fills up
That is what I want to say. It will cause consumers to be unable to receive messages that were sent to the new partitions until the client application restarts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "poorbarcode (via GitHub)" <gi...@apache.org>.
poorbarcode commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2492539745
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/TopicListService.java:
##########
@@ -101,6 +112,7 @@ public void accept(String topicName, NotificationType notificationType) {
private final boolean enableSubscriptionPatternEvaluation;
private final int maxSubscriptionPatternLength;
private final ConcurrentLongHashMap<CompletableFuture<TopicListWatcher>> watchers;
+ private final Backoff retryBackoff;
public TopicListService(PulsarService pulsar, ServerCnx connection,
Review Comment:
> unless it times out or the queue fills up
Thanks is what I want to say. It will cause consumers to be unable to receive messages that were sent to the new partitions until the client application restarts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2429735799
##########
pulsar-common/src/main/java/org/apache/pulsar/common/semaphore/AsyncSemaphoreImpl.java:
##########
@@ -0,0 +1,288 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.common.semaphore;
+
+import io.netty.util.concurrent.DefaultThreadFactory;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicLongFieldUpdater;
+import java.util.function.BooleanSupplier;
+import java.util.function.LongConsumer;
+import org.apache.pulsar.common.util.Runnables;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of AsyncSemaphore with timeout and queue size limits.
+ */
+public class AsyncSemaphoreImpl implements AsyncSemaphore, AutoCloseable {
+ private static final Logger log = LoggerFactory.getLogger(AsyncSemaphoreImpl.class);
+
+ private final AtomicLong availablePermits;
+ private final ConcurrentLinkedQueue<PendingRequest> queue = new ConcurrentLinkedQueue<>();
+ private final long maxPermits;
+ private final int maxQueueSize;
+ private final long timeoutMillis;
+ private final ScheduledExecutorService executor;
+ private final boolean shutdownExecutor;
+ private final LongConsumer queueLatencyRecorder;
+ private final AtomicBoolean closed = new AtomicBoolean(false);
+ private final Runnable processQueueRunnable = Runnables.catchingAndLoggingThrowables(this::internalProcessQueue);
+
+ public AsyncSemaphoreImpl(long maxPermits, int maxQueueSize, long timeoutMillis) {
+ this(maxPermits, maxQueueSize, timeoutMillis, createExecutor(), true, null);
+ }
+
+ public AsyncSemaphoreImpl(long maxPermits, int maxQueueSize, long timeoutMillis,
+ ScheduledExecutorService executor, LongConsumer queueLatencyRecorder) {
+ this(maxPermits, maxQueueSize, timeoutMillis, executor, false, queueLatencyRecorder);
+ }
+
+ AsyncSemaphoreImpl(long maxPermits, int maxQueueSize, long timeoutMillis, ScheduledExecutorService executor,
+ boolean shutdownExecutor, LongConsumer queueLatencyRecorder) {
+ this.availablePermits = new AtomicLong(maxPermits);
+ this.maxPermits = maxPermits;
+ this.maxQueueSize = maxQueueSize;
+ this.timeoutMillis = timeoutMillis;
+ this.executor = executor;
+ this.shutdownExecutor = shutdownExecutor;
+ this.queueLatencyRecorder = queueLatencyRecorder;
+ }
+
+ private static ScheduledExecutorService createExecutor() {
+ return Executors.newSingleThreadScheduledExecutor(
+ new DefaultThreadFactory("async-semaphore-executor"));
+ }
+
+ @Override
+ public CompletableFuture<AsyncSemaphorePermit> acquire(long permits, BooleanSupplier isCancelled) {
+ return internalAcquire(permits, permits, isCancelled);
+ }
+
+ private CompletableFuture<AsyncSemaphorePermit> internalAcquire(long permits, long acquirePermits,
+ BooleanSupplier isCancelled) {
+ if (permits < 0) {
+ throw new IllegalArgumentException("Invalid permits value: " + permits);
+ }
+
+ CompletableFuture<AsyncSemaphorePermit> future = new CompletableFuture<>();
+
+ if (closed.get()) {
+ future.completeExceptionally(new PermitAcquireAlreadyClosedException("Semaphore is closed"));
+ return future;
+ }
+
+ if (queue.size() >= maxQueueSize) {
Review Comment:
I'll replace the queue implementation with ArrayBlockingQueue and use `offer` to add to the queue. (it returns false if the queue size limit has been reached)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2429775675
##########
pulsar-broker-common/src/main/java/org/apache/pulsar/broker/topiclistlimit/TopicListMemoryLimiter.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.topiclistlimit;
+
+import io.opentelemetry.api.metrics.DoubleGauge;
+import io.opentelemetry.api.metrics.DoubleHistogram;
+import io.opentelemetry.api.metrics.LongCounter;
+import io.opentelemetry.api.metrics.LongGauge;
+import io.opentelemetry.api.metrics.Meter;
+import io.opentelemetry.api.metrics.ObservableDoubleGauge;
+import io.opentelemetry.api.metrics.ObservableLongUpDownCounter;
+import io.prometheus.client.Collector;
+import io.prometheus.client.CollectorRegistry;
+import io.prometheus.client.Counter;
+import io.prometheus.client.Gauge;
+import io.prometheus.client.Summary;
+import java.util.concurrent.TimeUnit;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.common.semaphore.AsyncDualMemoryLimiterImpl;
+import org.apache.pulsar.common.semaphore.AsyncSemaphore;
+
+/**
+ * Topic list memory limiter that exposes both Prometheus metrics and OpenTelemetry metrics.
+ */
+@Slf4j
+public class TopicListMemoryLimiter extends AsyncDualMemoryLimiterImpl {
+ private final CollectorRegistry collectorRegistry;
+ private final Gauge heapMemoryUsedBytes;
+ private final Gauge heapMemoryLimitBytes;
+ private final Gauge directMemoryUsedBytes;
+ private final Gauge directMemoryLimitBytes;
+ private final Gauge heapQueueSize;
+ private final Gauge heapQueueMaxSize;
+ private final Gauge directQueueSize;
+ private final Gauge directQueueMaxSize;
+ private final Summary heapWaitTimeMs;
+ private final Summary directWaitTimeMs;
+ private final Counter heapTimeoutTotal;
+ private final Counter directTimeoutTotal;
+ private final ObservableDoubleGauge otelHeapMemoryUsedGauge;
+ private final DoubleGauge otelHeapMemoryLimitGauge;
+ private final ObservableDoubleGauge otelDirectMemoryUsedGauge;
+ private final DoubleGauge otelDirectMemoryLimitGauge;
+ private final ObservableLongUpDownCounter otelHeapQueueSize;
+ private final ObservableLongUpDownCounter otelDirectQueueSize;
+ private final DoubleHistogram otelHeapWaitTime;
+ private final DoubleHistogram otelDirectWaitTime;
+ private final LongCounter otelHeapTimeoutTotal;
+ private final LongCounter otelDirectTimeoutTotal;
+
+ public TopicListMemoryLimiter(CollectorRegistry collectorRegistry, String prometheusPrefix,
+ Meter openTelemetryMeter,
+ long maxHeapMemory, int maxHeapQueueSize,
+ long heapTimeoutMillis, long maxDirectMemory, int maxDirectQueueSize,
+ long directTimeoutMillis) {
+ super(maxHeapMemory, maxHeapQueueSize, heapTimeoutMillis, maxDirectMemory, maxDirectQueueSize,
+ directTimeoutMillis);
+ this.collectorRegistry = collectorRegistry;
+
+ AsyncSemaphore heapMemoryLimiter = getLimiter(LimitType.HEAP_MEMORY);
+ AsyncSemaphore directMemoryLimiter = getLimiter(LimitType.DIRECT_MEMORY);
+
+ this.heapMemoryUsedBytes = register(Gauge.build(prometheusPrefix + "topic_list_heap_memory_used_bytes",
+ "Current heap memory used by topic listings")
+ .create()
+ .setChild(new Gauge.Child() {
+ @Override
+ public double get() {
+ return heapMemoryLimiter.getAcquiredPermits();
+ }
+ }));
+ this.otelHeapMemoryUsedGauge = openTelemetryMeter.gaugeBuilder("topic.list.heap.memory.used")
+ .setUnit("By")
+ .setDescription("Current heap memory used by topic listings")
+ .buildWithCallback(observableDoubleMeasurement -> {
+ observableDoubleMeasurement.record(heapMemoryLimiter.getAcquiredPermits());
+ });
+
+ this.heapMemoryLimitBytes = register(Gauge.build(prometheusPrefix + "topic_list_heap_memory_limit_bytes",
+ "Configured heap memory limit")
+ .create());
+ this.heapMemoryLimitBytes.set(maxHeapMemory);
+ this.otelHeapMemoryLimitGauge = openTelemetryMeter.gaugeBuilder("topic.list.heap.memory.limit")
+ .setUnit("By")
+ .setDescription("Configured heap memory limit")
+ .build();
+ this.otelHeapMemoryLimitGauge.set(maxHeapMemory);
+
+ this.directMemoryUsedBytes = register(Gauge.build(prometheusPrefix + "topic_list_direct_memory_used_bytes",
+ "Current direct memory used by topic listings")
+ .create()
+ .setChild(new Gauge.Child() {
+ @Override
+ public double get() {
+ return directMemoryLimiter.getAcquiredPermits();
+ }
+ }));
+ this.otelDirectMemoryUsedGauge = openTelemetryMeter.gaugeBuilder("topic.list.direct.memory.used")
+ .setUnit("By")
+ .setDescription("Current direct memory used by topic listings")
+ .buildWithCallback(observableDoubleMeasurement -> {
+ observableDoubleMeasurement.record(directMemoryLimiter.getAcquiredPermits());
+ });
+
+ this.directMemoryLimitBytes = register(Gauge.build(prometheusPrefix + "topic_list_direct_memory_limit_bytes",
+ "Configured direct memory limit")
+ .create());
+ this.directMemoryLimitBytes.set(maxDirectMemory);
+ this.otelDirectMemoryLimitGauge = openTelemetryMeter.gaugeBuilder("topic.list.direct.memory.limit")
+ .setUnit("By")
+ .setDescription("Configured direct memory limit")
+ .build();
+ this.otelDirectMemoryLimitGauge.set(maxDirectMemory);
+
+ this.heapQueueSize = register(Gauge.build(prometheusPrefix + "topic_list_heap_queue_size",
+ "Current heap memory limiter queue size")
+ .create()
+ .setChild(new Gauge.Child() {
+ @Override
+ public double get() {
+ return heapMemoryLimiter.getQueueSize();
+ }
+ }));
+ this.otelHeapQueueSize = openTelemetryMeter
+ .upDownCounterBuilder("topic.list.heap.queue.size")
+ .setDescription("Current heap memory limiter queue size")
+ .setUnit("1")
+ .buildWithCallback(observableLongMeasurement -> {
+ observableLongMeasurement.record(heapMemoryLimiter.getQueueSize());
+ });
+
+ this.heapQueueMaxSize = register(Gauge.build(prometheusPrefix + "topic_list_heap_queue_max_size",
+ "Maximum heap memory limiter queue size")
+ .create());
+ this.heapQueueMaxSize.set(maxHeapQueueSize);
+ LongGauge otelHeapQueueMaxSize = openTelemetryMeter
+ .gaugeBuilder("topic.list.heap.queue.max.size")
+ .setDescription("Maximum heap memory limiter queue size")
+ .setUnit("1")
+ .ofLongs()
+ .build();
+ otelHeapQueueMaxSize.set(maxHeapQueueSize);
Review Comment:
there is no close method in Otel for instruments that aren't "observable".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "Denovo1998 (via GitHub)" <gi...@apache.org>.
Denovo1998 commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2429175692
##########
pulsar-broker-common/src/main/java/org/apache/pulsar/broker/topiclistlimit/TopicListMemoryLimiter.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.topiclistlimit;
+
+import io.opentelemetry.api.metrics.DoubleGauge;
+import io.opentelemetry.api.metrics.DoubleHistogram;
+import io.opentelemetry.api.metrics.LongCounter;
+import io.opentelemetry.api.metrics.LongGauge;
+import io.opentelemetry.api.metrics.Meter;
+import io.opentelemetry.api.metrics.ObservableDoubleGauge;
+import io.opentelemetry.api.metrics.ObservableLongUpDownCounter;
+import io.prometheus.client.Collector;
+import io.prometheus.client.CollectorRegistry;
+import io.prometheus.client.Counter;
+import io.prometheus.client.Gauge;
+import io.prometheus.client.Summary;
+import java.util.concurrent.TimeUnit;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.common.semaphore.AsyncDualMemoryLimiterImpl;
+import org.apache.pulsar.common.semaphore.AsyncSemaphore;
+
+/**
+ * Topic list memory limiter that exposes both Prometheus metrics and OpenTelemetry metrics.
+ */
+@Slf4j
+public class TopicListMemoryLimiter extends AsyncDualMemoryLimiterImpl {
+ private final CollectorRegistry collectorRegistry;
+ private final Gauge heapMemoryUsedBytes;
+ private final Gauge heapMemoryLimitBytes;
+ private final Gauge directMemoryUsedBytes;
+ private final Gauge directMemoryLimitBytes;
+ private final Gauge heapQueueSize;
+ private final Gauge heapQueueMaxSize;
+ private final Gauge directQueueSize;
+ private final Gauge directQueueMaxSize;
+ private final Summary heapWaitTimeMs;
+ private final Summary directWaitTimeMs;
+ private final Counter heapTimeoutTotal;
+ private final Counter directTimeoutTotal;
+ private final ObservableDoubleGauge otelHeapMemoryUsedGauge;
+ private final DoubleGauge otelHeapMemoryLimitGauge;
+ private final ObservableDoubleGauge otelDirectMemoryUsedGauge;
+ private final DoubleGauge otelDirectMemoryLimitGauge;
+ private final ObservableLongUpDownCounter otelHeapQueueSize;
+ private final ObservableLongUpDownCounter otelDirectQueueSize;
+ private final DoubleHistogram otelHeapWaitTime;
+ private final DoubleHistogram otelDirectWaitTime;
+ private final LongCounter otelHeapTimeoutTotal;
+ private final LongCounter otelDirectTimeoutTotal;
+
+ public TopicListMemoryLimiter(CollectorRegistry collectorRegistry, String prometheusPrefix,
+ Meter openTelemetryMeter,
+ long maxHeapMemory, int maxHeapQueueSize,
+ long heapTimeoutMillis, long maxDirectMemory, int maxDirectQueueSize,
+ long directTimeoutMillis) {
+ super(maxHeapMemory, maxHeapQueueSize, heapTimeoutMillis, maxDirectMemory, maxDirectQueueSize,
+ directTimeoutMillis);
+ this.collectorRegistry = collectorRegistry;
+
+ AsyncSemaphore heapMemoryLimiter = getLimiter(LimitType.HEAP_MEMORY);
+ AsyncSemaphore directMemoryLimiter = getLimiter(LimitType.DIRECT_MEMORY);
+
+ this.heapMemoryUsedBytes = register(Gauge.build(prometheusPrefix + "topic_list_heap_memory_used_bytes",
+ "Current heap memory used by topic listings")
+ .create()
+ .setChild(new Gauge.Child() {
+ @Override
+ public double get() {
+ return heapMemoryLimiter.getAcquiredPermits();
+ }
+ }));
+ this.otelHeapMemoryUsedGauge = openTelemetryMeter.gaugeBuilder("topic.list.heap.memory.used")
+ .setUnit("By")
+ .setDescription("Current heap memory used by topic listings")
+ .buildWithCallback(observableDoubleMeasurement -> {
+ observableDoubleMeasurement.record(heapMemoryLimiter.getAcquiredPermits());
+ });
+
+ this.heapMemoryLimitBytes = register(Gauge.build(prometheusPrefix + "topic_list_heap_memory_limit_bytes",
+ "Configured heap memory limit")
+ .create());
+ this.heapMemoryLimitBytes.set(maxHeapMemory);
+ this.otelHeapMemoryLimitGauge = openTelemetryMeter.gaugeBuilder("topic.list.heap.memory.limit")
+ .setUnit("By")
+ .setDescription("Configured heap memory limit")
+ .build();
+ this.otelHeapMemoryLimitGauge.set(maxHeapMemory);
+
+ this.directMemoryUsedBytes = register(Gauge.build(prometheusPrefix + "topic_list_direct_memory_used_bytes",
+ "Current direct memory used by topic listings")
+ .create()
+ .setChild(new Gauge.Child() {
+ @Override
+ public double get() {
+ return directMemoryLimiter.getAcquiredPermits();
+ }
+ }));
+ this.otelDirectMemoryUsedGauge = openTelemetryMeter.gaugeBuilder("topic.list.direct.memory.used")
+ .setUnit("By")
+ .setDescription("Current direct memory used by topic listings")
+ .buildWithCallback(observableDoubleMeasurement -> {
+ observableDoubleMeasurement.record(directMemoryLimiter.getAcquiredPermits());
+ });
+
+ this.directMemoryLimitBytes = register(Gauge.build(prometheusPrefix + "topic_list_direct_memory_limit_bytes",
+ "Configured direct memory limit")
+ .create());
+ this.directMemoryLimitBytes.set(maxDirectMemory);
+ this.otelDirectMemoryLimitGauge = openTelemetryMeter.gaugeBuilder("topic.list.direct.memory.limit")
+ .setUnit("By")
+ .setDescription("Configured direct memory limit")
+ .build();
+ this.otelDirectMemoryLimitGauge.set(maxDirectMemory);
+
+ this.heapQueueSize = register(Gauge.build(prometheusPrefix + "topic_list_heap_queue_size",
+ "Current heap memory limiter queue size")
+ .create()
+ .setChild(new Gauge.Child() {
+ @Override
+ public double get() {
+ return heapMemoryLimiter.getQueueSize();
+ }
+ }));
+ this.otelHeapQueueSize = openTelemetryMeter
+ .upDownCounterBuilder("topic.list.heap.queue.size")
+ .setDescription("Current heap memory limiter queue size")
+ .setUnit("1")
+ .buildWithCallback(observableLongMeasurement -> {
+ observableLongMeasurement.record(heapMemoryLimiter.getQueueSize());
+ });
+
+ this.heapQueueMaxSize = register(Gauge.build(prometheusPrefix + "topic_list_heap_queue_max_size",
+ "Maximum heap memory limiter queue size")
+ .create());
+ this.heapQueueMaxSize.set(maxHeapQueueSize);
+ LongGauge otelHeapQueueMaxSize = openTelemetryMeter
+ .gaugeBuilder("topic.list.heap.queue.max.size")
+ .setDescription("Maximum heap memory limiter queue size")
+ .setUnit("1")
+ .ofLongs()
+ .build();
+ otelHeapQueueMaxSize.set(maxHeapQueueSize);
+
+ this.directQueueSize = register(Gauge.build(prometheusPrefix + "topic_list_direct_queue_size",
+ "Current direct memory limiter queue size")
+ .create()
+ .setChild(new Gauge.Child() {
+ @Override
+ public double get() {
+ return directMemoryLimiter.getQueueSize();
+ }
+ }));
+ this.otelDirectQueueSize = openTelemetryMeter
+ .upDownCounterBuilder("topic.list.direct.queue.size")
+ .setDescription("Current direct memory limiter queue size")
+ .setUnit("1")
+ .buildWithCallback(observableLongMeasurement -> {
+ observableLongMeasurement.record(directMemoryLimiter.getQueueSize());
+ });
+
+ this.directQueueMaxSize = register(Gauge.build(prometheusPrefix + "topic_list_direct_queue_max_size",
+ "Maximum direct memory limiter queue size")
+ .create());
+ this.directQueueMaxSize.set(maxDirectQueueSize);
+ LongGauge otelDirectQueueMaxSize = openTelemetryMeter
+ .gaugeBuilder("topic.list.direct.queue.max.size")
+ .setDescription("Maximum direct memory limiter queue size")
+ .setUnit("1")
+ .ofLongs()
+ .build();
+ otelDirectQueueMaxSize.set(maxDirectQueueSize);
+
+ this.heapWaitTimeMs = register(Summary.build(prometheusPrefix + "topic_list_heap_wait_time_ms",
+ "Wait time for heap memory permits")
+ .quantile(0.50, 0.01)
+ .quantile(0.95, 0.01)
+ .quantile(0.99, 0.01)
+ .quantile(1, 0.01)
+ .create());
+ this.otelHeapWaitTime = openTelemetryMeter.histogramBuilder("topic.list.heap.wait.time.ms")
+ .setUnit("s")
+ .setDescription("Wait time for heap memory permits")
+ .build();
+
+ this.directWaitTimeMs = register(Summary.build(prometheusPrefix + "topic_list_direct_wait_time_ms",
+ "Wait time for direct memory permits")
+ .quantile(0.50, 0.01)
+ .quantile(0.95, 0.01)
+ .quantile(0.99, 0.01)
+ .quantile(1, 0.01)
+ .create());
+ this.otelDirectWaitTime = openTelemetryMeter.histogramBuilder("topic.list.direct.wait.time.ms")
+ .setUnit("s")
+ .setDescription("Wait time for direct memory permits")
+ .build();
+
+ this.heapTimeoutTotal = register(Counter.build(prometheusPrefix + "topic_list_heap_timeout_total",
+ "Total heap memory permit timeouts")
+ .create());
+ this.otelHeapTimeoutTotal = openTelemetryMeter.counterBuilder("topic.list.heap.timeout.total")
+ .setDescription("Total heap memory permit timeouts")
+ .setUnit("1")
+ .build();
+
+ this.directTimeoutTotal = register(Counter.build(prometheusPrefix + "topic_list_direct_timeout_total",
+ "Total direct memory permit timeouts")
+ .create());
+ this.otelDirectTimeoutTotal = openTelemetryMeter.counterBuilder("topic.list.direct.timeout.total")
+ .setDescription("Total direct memory permit timeouts")
+ .setUnit("1")
+ .build();
+ }
+
+ private <T extends Collector> T register(T collector) {
+ try {
+ collectorRegistry.register(collector);
+ } catch (Exception e) {
+ // ignore exception when registering a collector that is already registered
+ if (log.isDebugEnabled()) {
+ log.debug("Failed to register Prometheus collector {}", collector, e);
+ }
+ }
+ return collector;
+ }
+
+ private void unregister(Collector collector) {
+ try {
+ collectorRegistry.unregister(collector);
+ } catch (Exception e) {
+ // ignore exception when unregistering a collector that is not registered
+ if (log.isDebugEnabled()) {
+ log.debug("Failed to unregister Prometheus collector {}", collector, e);
+ }
+ }
+ }
+
+
+ @Override
+ protected void recordHeapWaitTime(long waitTimeNanos) {
+ if (waitTimeNanos == Long.MAX_VALUE) {
+ heapTimeoutTotal.inc();
+ otelHeapTimeoutTotal.add(1);
+ } else {
+ heapWaitTimeMs.observe(TimeUnit.NANOSECONDS.toMillis(waitTimeNanos));
+ otelHeapWaitTime.record(waitTimeNanos / 1_000_000_000.0d);
+ }
+ }
+
+ @Override
+ protected void recordDirectWaitTime(long waitTimeNanos) {
+ if (waitTimeNanos == Long.MAX_VALUE) {
+ directTimeoutTotal.inc();
+ otelDirectTimeoutTotal.add(1);
+ } else {
+ directWaitTimeMs.observe(TimeUnit.NANOSECONDS.toMillis(waitTimeNanos));
+ otelDirectWaitTime.record(waitTimeNanos / 1_000_000_000.0d);
+ }
+ }
+
+ @Override
+ public void close() {
+ super.close();
+ unregister(heapMemoryUsedBytes);
+ unregister(heapMemoryLimitBytes);
+ unregister(directMemoryUsedBytes);
+ unregister(directMemoryLimitBytes);
+ unregister(heapQueueSize);
+ unregister(heapQueueMaxSize);
+ unregister(directQueueSize);
+ unregister(directQueueMaxSize);
+ unregister(heapWaitTimeMs);
+ unregister(directWaitTimeMs);
+ unregister(heapTimeoutTotal);
+ unregister(directTimeoutTotal);
+ otelHeapMemoryUsedGauge.close();
+ otelDirectMemoryUsedGauge.close();
+ otelHeapQueueSize.close();
Review Comment:
Close all OpenTelemetry instruments to prevent resource leaks.
```java
otelHeapMemoryLimitGauge.close();
otelDirectMemoryLimitGauge.close();
otelHeapQueueMaxSize.close();
otelDirectQueueMaxSize.close();
otelHeapWaitTime.close();
otelDirectWaitTime.close();
otelHeapTimeoutTotal.close();
otelDirectTimeoutTotal.close();
```
##########
pulsar-broker-common/src/main/java/org/apache/pulsar/broker/topiclistlimit/TopicListMemoryLimiter.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.topiclistlimit;
+
+import io.opentelemetry.api.metrics.DoubleGauge;
+import io.opentelemetry.api.metrics.DoubleHistogram;
+import io.opentelemetry.api.metrics.LongCounter;
+import io.opentelemetry.api.metrics.LongGauge;
+import io.opentelemetry.api.metrics.Meter;
+import io.opentelemetry.api.metrics.ObservableDoubleGauge;
+import io.opentelemetry.api.metrics.ObservableLongUpDownCounter;
+import io.prometheus.client.Collector;
+import io.prometheus.client.CollectorRegistry;
+import io.prometheus.client.Counter;
+import io.prometheus.client.Gauge;
+import io.prometheus.client.Summary;
+import java.util.concurrent.TimeUnit;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.common.semaphore.AsyncDualMemoryLimiterImpl;
+import org.apache.pulsar.common.semaphore.AsyncSemaphore;
+
+/**
+ * Topic list memory limiter that exposes both Prometheus metrics and OpenTelemetry metrics.
+ */
+@Slf4j
+public class TopicListMemoryLimiter extends AsyncDualMemoryLimiterImpl {
+ private final CollectorRegistry collectorRegistry;
+ private final Gauge heapMemoryUsedBytes;
+ private final Gauge heapMemoryLimitBytes;
+ private final Gauge directMemoryUsedBytes;
+ private final Gauge directMemoryLimitBytes;
+ private final Gauge heapQueueSize;
+ private final Gauge heapQueueMaxSize;
+ private final Gauge directQueueSize;
+ private final Gauge directQueueMaxSize;
+ private final Summary heapWaitTimeMs;
+ private final Summary directWaitTimeMs;
+ private final Counter heapTimeoutTotal;
+ private final Counter directTimeoutTotal;
+ private final ObservableDoubleGauge otelHeapMemoryUsedGauge;
+ private final DoubleGauge otelHeapMemoryLimitGauge;
+ private final ObservableDoubleGauge otelDirectMemoryUsedGauge;
+ private final DoubleGauge otelDirectMemoryLimitGauge;
+ private final ObservableLongUpDownCounter otelHeapQueueSize;
+ private final ObservableLongUpDownCounter otelDirectQueueSize;
+ private final DoubleHistogram otelHeapWaitTime;
+ private final DoubleHistogram otelDirectWaitTime;
+ private final LongCounter otelHeapTimeoutTotal;
+ private final LongCounter otelDirectTimeoutTotal;
+
+ public TopicListMemoryLimiter(CollectorRegistry collectorRegistry, String prometheusPrefix,
+ Meter openTelemetryMeter,
+ long maxHeapMemory, int maxHeapQueueSize,
+ long heapTimeoutMillis, long maxDirectMemory, int maxDirectQueueSize,
+ long directTimeoutMillis) {
+ super(maxHeapMemory, maxHeapQueueSize, heapTimeoutMillis, maxDirectMemory, maxDirectQueueSize,
+ directTimeoutMillis);
+ this.collectorRegistry = collectorRegistry;
+
+ AsyncSemaphore heapMemoryLimiter = getLimiter(LimitType.HEAP_MEMORY);
+ AsyncSemaphore directMemoryLimiter = getLimiter(LimitType.DIRECT_MEMORY);
+
+ this.heapMemoryUsedBytes = register(Gauge.build(prometheusPrefix + "topic_list_heap_memory_used_bytes",
+ "Current heap memory used by topic listings")
+ .create()
+ .setChild(new Gauge.Child() {
+ @Override
+ public double get() {
+ return heapMemoryLimiter.getAcquiredPermits();
+ }
+ }));
+ this.otelHeapMemoryUsedGauge = openTelemetryMeter.gaugeBuilder("topic.list.heap.memory.used")
+ .setUnit("By")
+ .setDescription("Current heap memory used by topic listings")
+ .buildWithCallback(observableDoubleMeasurement -> {
+ observableDoubleMeasurement.record(heapMemoryLimiter.getAcquiredPermits());
+ });
+
+ this.heapMemoryLimitBytes = register(Gauge.build(prometheusPrefix + "topic_list_heap_memory_limit_bytes",
+ "Configured heap memory limit")
+ .create());
+ this.heapMemoryLimitBytes.set(maxHeapMemory);
+ this.otelHeapMemoryLimitGauge = openTelemetryMeter.gaugeBuilder("topic.list.heap.memory.limit")
+ .setUnit("By")
+ .setDescription("Configured heap memory limit")
+ .build();
+ this.otelHeapMemoryLimitGauge.set(maxHeapMemory);
+
+ this.directMemoryUsedBytes = register(Gauge.build(prometheusPrefix + "topic_list_direct_memory_used_bytes",
+ "Current direct memory used by topic listings")
+ .create()
+ .setChild(new Gauge.Child() {
+ @Override
+ public double get() {
+ return directMemoryLimiter.getAcquiredPermits();
+ }
+ }));
+ this.otelDirectMemoryUsedGauge = openTelemetryMeter.gaugeBuilder("topic.list.direct.memory.used")
+ .setUnit("By")
+ .setDescription("Current direct memory used by topic listings")
+ .buildWithCallback(observableDoubleMeasurement -> {
+ observableDoubleMeasurement.record(directMemoryLimiter.getAcquiredPermits());
+ });
+
+ this.directMemoryLimitBytes = register(Gauge.build(prometheusPrefix + "topic_list_direct_memory_limit_bytes",
+ "Configured direct memory limit")
+ .create());
+ this.directMemoryLimitBytes.set(maxDirectMemory);
+ this.otelDirectMemoryLimitGauge = openTelemetryMeter.gaugeBuilder("topic.list.direct.memory.limit")
+ .setUnit("By")
+ .setDescription("Configured direct memory limit")
+ .build();
+ this.otelDirectMemoryLimitGauge.set(maxDirectMemory);
+
+ this.heapQueueSize = register(Gauge.build(prometheusPrefix + "topic_list_heap_queue_size",
+ "Current heap memory limiter queue size")
+ .create()
+ .setChild(new Gauge.Child() {
+ @Override
+ public double get() {
+ return heapMemoryLimiter.getQueueSize();
+ }
+ }));
+ this.otelHeapQueueSize = openTelemetryMeter
+ .upDownCounterBuilder("topic.list.heap.queue.size")
+ .setDescription("Current heap memory limiter queue size")
+ .setUnit("1")
+ .buildWithCallback(observableLongMeasurement -> {
+ observableLongMeasurement.record(heapMemoryLimiter.getQueueSize());
+ });
+
+ this.heapQueueMaxSize = register(Gauge.build(prometheusPrefix + "topic_list_heap_queue_max_size",
+ "Maximum heap memory limiter queue size")
+ .create());
+ this.heapQueueMaxSize.set(maxHeapQueueSize);
+ LongGauge otelHeapQueueMaxSize = openTelemetryMeter
+ .gaugeBuilder("topic.list.heap.queue.max.size")
+ .setDescription("Maximum heap memory limiter queue size")
+ .setUnit("1")
+ .ofLongs()
+ .build();
+ otelHeapQueueMaxSize.set(maxHeapQueueSize);
+
+ this.directQueueSize = register(Gauge.build(prometheusPrefix + "topic_list_direct_queue_size",
+ "Current direct memory limiter queue size")
+ .create()
+ .setChild(new Gauge.Child() {
+ @Override
+ public double get() {
+ return directMemoryLimiter.getQueueSize();
+ }
+ }));
+ this.otelDirectQueueSize = openTelemetryMeter
+ .upDownCounterBuilder("topic.list.direct.queue.size")
+ .setDescription("Current direct memory limiter queue size")
+ .setUnit("1")
+ .buildWithCallback(observableLongMeasurement -> {
+ observableLongMeasurement.record(directMemoryLimiter.getQueueSize());
+ });
+
+ this.directQueueMaxSize = register(Gauge.build(prometheusPrefix + "topic_list_direct_queue_max_size",
+ "Maximum direct memory limiter queue size")
+ .create());
+ this.directQueueMaxSize.set(maxDirectQueueSize);
+ LongGauge otelDirectQueueMaxSize = openTelemetryMeter
+ .gaugeBuilder("topic.list.direct.queue.max.size")
+ .setDescription("Maximum direct memory limiter queue size")
+ .setUnit("1")
+ .ofLongs()
+ .build();
+ otelDirectQueueMaxSize.set(maxDirectQueueSize);
+
+ this.heapWaitTimeMs = register(Summary.build(prometheusPrefix + "topic_list_heap_wait_time_ms",
+ "Wait time for heap memory permits")
+ .quantile(0.50, 0.01)
+ .quantile(0.95, 0.01)
+ .quantile(0.99, 0.01)
+ .quantile(1, 0.01)
+ .create());
+ this.otelHeapWaitTime = openTelemetryMeter.histogramBuilder("topic.list.heap.wait.time.ms")
+ .setUnit("s")
+ .setDescription("Wait time for heap memory permits")
+ .build();
+
+ this.directWaitTimeMs = register(Summary.build(prometheusPrefix + "topic_list_direct_wait_time_ms",
+ "Wait time for direct memory permits")
+ .quantile(0.50, 0.01)
+ .quantile(0.95, 0.01)
+ .quantile(0.99, 0.01)
+ .quantile(1, 0.01)
+ .create());
+ this.otelDirectWaitTime = openTelemetryMeter.histogramBuilder("topic.list.direct.wait.time.ms")
+ .setUnit("s")
Review Comment:
The Prometheus metric name uses _wait_time_ms (milliseconds) but the OpenTelemetry histogram unit is set to "s" (seconds).
While the actual recording in recordDirectWaitTime correctly converts to seconds, the metric name .wait.time.ms suggests milliseconds, which is confusing.
##########
pulsar-common/src/main/java/org/apache/pulsar/common/semaphore/AsyncSemaphore.java:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.common.semaphore;
+
+import java.util.concurrent.CompletableFuture;
+import java.util.function.BooleanSupplier;
+
+/**
+ * An abstraction for a generic asynchronous semaphore.
+ */
+public interface AsyncSemaphore {
+ /**
+ * Acquire permits from the semaphore.
+ * Returned future completes when permits are available.
+ * It will complete exceptionally with AsyncSemaphorePermitAcquireTimeoutException on timeout
Review Comment:
typo: `AsyncSemaphore.PermitAcquireTimeoutException`
##########
pulsar-broker/src/test/java/org/apache/pulsar/client/api/PatternConsumerBackPressureMultipleConsumersTest.java:
##########
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.client.api;
+
+import static org.testng.Assert.assertEquals;
+import io.netty.buffer.ByteBuf;
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.UUID;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.Cleanup;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.mutable.MutableInt;
+import org.apache.pulsar.broker.auth.MockedPulsarServiceBaseTest;
+import org.apache.pulsar.broker.stats.prometheus.PrometheusMetricsClient;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.allocator.PulsarByteBufAllocator;
+import org.apache.pulsar.common.api.proto.CommandGetTopicsOfNamespace;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.stats.JvmMetrics;
+import org.apache.pulsar.common.util.DirectMemoryUtils;
+import org.testng.annotations.AfterMethod;
+import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.Test;
+
+@Slf4j
+@Test(groups = "broker-impl")
+public class PatternConsumerBackPressureMultipleConsumersTest extends MockedPulsarServiceBaseTest {
+
+ @Override
+ @BeforeMethod
+ protected void setup() throws Exception {
+ isTcpLookup = true;
+ super.internalSetup();
+ setupDefaultTenantAndNamespace();
+ }
+
+ @Override
+ @AfterMethod(alwaysRun = true)
+ protected void cleanup() throws Exception {
+ super.internalCleanup();
+ }
+
+ @Test(timeOut = 60 * 1000)
+ public void testGetTopicsWithLargeAmountOfConcurrentClientConnections()
+ throws PulsarAdminException, InterruptedException, IOException {
+ // number of requests to send to the broker
+ final int requests = 500;
+ // use multiple clients so that each client has a separate connection to the broker
+ final int numberOfClients = 200;
+ // create a long topic name to consume more memory per topic
+ final String topicName = StringUtils.repeat('a', 512) + UUID.randomUUID();
+ // number of topics to create
+ final int topicCount = 8192;
+ // maximum number of requests in flight at any given time
+ final int maxRequestsInFlight = 200;
+
+ // create a single topic with multiple partitions
+ admin.topics().createPartitionedTopic(topicName, topicCount);
+
+ // reduce available direct memory to reproduce issues with less concurrency
+ long directMemoryRequired = getDirectMemoryRequiredMB() * 1024 * 1024;
+ List<ByteBuf> buffers = allocateDirectMemory(directMemoryRequired);
+ @Cleanup
+ Closeable releaseBuffers = () -> {
+ for (ByteBuf byteBuf : buffers) {
+ byteBuf.release();
+ }
+ };
+
+ @Cleanup("shutdownNow")
+ final ExecutorService executorService = Executors.newFixedThreadPool(Runtime.getRuntime()
+ .availableProcessors());
+
+ @Cleanup
+ PulsarClientSharedResources sharedResources =
+ PulsarClientSharedResources.builder().build();
+ List<PulsarClientImpl> clients = new ArrayList<>(requests);
+ @Cleanup
+ Closeable closeClients = () -> {
+ for (PulsarClient client : clients) {
+ try {
+ client.close();
+ } catch (PulsarClientException e) {
+ log.error("Failed to close client {}", client, e);
+ }
+ }
+ };
+ for (int i = 0; i < numberOfClients; i++) {
+ PulsarClientImpl client = (PulsarClientImpl) PulsarClient.builder()
+ .serviceUrl(getClientServiceUrl())
+ .sharedResources(sharedResources)
+ .build();
+ clients.add(client);
+ }
+
+ final AtomicInteger success = new AtomicInteger(0);
+ final CountDownLatch latch = new CountDownLatch(requests);
+ final Semaphore semaphore = new Semaphore(maxRequestsInFlight);
+ for (int i = 0; i < requests; i++) {
+ PulsarClientImpl pulsarClientImpl = clients.get(i % numberOfClients);
+ executorService.execute(() -> {
+ semaphore.acquireUninterruptibly();
+ try {
+ pulsarClientImpl.getLookup()
+ .getTopicsUnderNamespace(NamespaceName.get("public", "default"),
+ CommandGetTopicsOfNamespace.Mode.PERSISTENT, ".*", "")
+ .whenComplete((result, ex) -> {
+ semaphore.release();
+ if (ex == null) {
+ success.incrementAndGet();
+ } else {
+ log.error("Failed to get topic list.", ex);
+ }
+ log.info("latch-count: {}, succeed: {}, available direct mem: {} MB", latch.getCount(),
+ success.get(),
+ (DirectMemoryUtils.jvmMaxDirectMemory() - JvmMetrics.getJvmDirectMemoryUsed())
+ / (1024 * 1024));
+ latch.countDown();
+ });
+ } catch (Exception e) {
+ semaphore.release();
Review Comment:
Should add this?
```java
latch.countDown();
log.error("Failed to execute getTopicsUnderNamespace request.", e);
```
##########
pulsar-common/src/main/java/org/apache/pulsar/common/semaphore/AsyncSemaphore.java:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.common.semaphore;
+
+import java.util.concurrent.CompletableFuture;
+import java.util.function.BooleanSupplier;
+
+/**
+ * An abstraction for a generic asynchronous semaphore.
+ */
+public interface AsyncSemaphore {
+ /**
+ * Acquire permits from the semaphore.
+ * Returned future completes when permits are available.
+ * It will complete exceptionally with AsyncSemaphorePermitAcquireTimeoutException on timeout
+ * and exceptionally with AsyncSemaphorePermitAcquireQueueFullException when queue full
+ *
+ * @param permits number of permits to acquire
+ * @param isCancelled supplier that returns true if acquisition should be cancelled
+ * @return CompletableFuture that completes with permit when available
+ */
+ CompletableFuture<AsyncSemaphorePermit> acquire(long permits, BooleanSupplier isCancelled);
+
+ /**
+ * Acquire or release permits for previously acquired permits by updating the permits.
+ * Returns a future that completes when permits are available.
+ * It will complete exceptionally with AsyncSemaphorePermitAcquireTimeoutException on timeout
+ * and exceptionally with AsyncSemaphorePermitAcquireQueueFullException when queue full
Review Comment:
typo: `AsyncSemaphore.PermitAcquireQueueFullException`
##########
pip/pip-442.md:
##########
@@ -132,24 +134,48 @@ public interface AsyncSemaphore {
* Returned future completes when permits are available.
* It will complete exceptionally with AsyncSemaphorePermitAcquireTimeoutException on timeout
* and exceptionally with AsyncSemaphorePermitAcquireQueueFullException when queue full
+ *
+ * @param permits number of permits to acquire
+ * @param isCancelled supplier that returns true if acquisition should be cancelled
* @return CompletableFuture that completes with permit when available
*/
- CompletableFuture<AsyncSemaphorePermitResult> acquire(long permits);
+ CompletableFuture<AsyncSemaphorePermit> acquire(long permits, BooleanSupplier isCancelled);
/**
* Acquire or release permits for previously acquired permits by updating the permits.
* Returns a future that completes when permits are available.
* It will complete exceptionally with AsyncSemaphorePermitAcquireTimeoutException on timeout
* and exceptionally with AsyncSemaphorePermitAcquireQueueFullException when queue full
Review Comment:
typo: `AsyncSemaphore.PermitAcquireQueueFullException`
##########
pulsar-broker-common/src/main/java/org/apache/pulsar/broker/topiclistlimit/TopicListMemoryLimiter.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.topiclistlimit;
+
+import io.opentelemetry.api.metrics.DoubleGauge;
+import io.opentelemetry.api.metrics.DoubleHistogram;
+import io.opentelemetry.api.metrics.LongCounter;
+import io.opentelemetry.api.metrics.LongGauge;
+import io.opentelemetry.api.metrics.Meter;
+import io.opentelemetry.api.metrics.ObservableDoubleGauge;
+import io.opentelemetry.api.metrics.ObservableLongUpDownCounter;
+import io.prometheus.client.Collector;
+import io.prometheus.client.CollectorRegistry;
+import io.prometheus.client.Counter;
+import io.prometheus.client.Gauge;
+import io.prometheus.client.Summary;
+import java.util.concurrent.TimeUnit;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.common.semaphore.AsyncDualMemoryLimiterImpl;
+import org.apache.pulsar.common.semaphore.AsyncSemaphore;
+
+/**
+ * Topic list memory limiter that exposes both Prometheus metrics and OpenTelemetry metrics.
+ */
+@Slf4j
+public class TopicListMemoryLimiter extends AsyncDualMemoryLimiterImpl {
+ private final CollectorRegistry collectorRegistry;
+ private final Gauge heapMemoryUsedBytes;
+ private final Gauge heapMemoryLimitBytes;
+ private final Gauge directMemoryUsedBytes;
+ private final Gauge directMemoryLimitBytes;
+ private final Gauge heapQueueSize;
+ private final Gauge heapQueueMaxSize;
+ private final Gauge directQueueSize;
+ private final Gauge directQueueMaxSize;
+ private final Summary heapWaitTimeMs;
+ private final Summary directWaitTimeMs;
+ private final Counter heapTimeoutTotal;
+ private final Counter directTimeoutTotal;
+ private final ObservableDoubleGauge otelHeapMemoryUsedGauge;
+ private final DoubleGauge otelHeapMemoryLimitGauge;
+ private final ObservableDoubleGauge otelDirectMemoryUsedGauge;
+ private final DoubleGauge otelDirectMemoryLimitGauge;
+ private final ObservableLongUpDownCounter otelHeapQueueSize;
+ private final ObservableLongUpDownCounter otelDirectQueueSize;
+ private final DoubleHistogram otelHeapWaitTime;
+ private final DoubleHistogram otelDirectWaitTime;
+ private final LongCounter otelHeapTimeoutTotal;
+ private final LongCounter otelDirectTimeoutTotal;
+
+ public TopicListMemoryLimiter(CollectorRegistry collectorRegistry, String prometheusPrefix,
+ Meter openTelemetryMeter,
+ long maxHeapMemory, int maxHeapQueueSize,
+ long heapTimeoutMillis, long maxDirectMemory, int maxDirectQueueSize,
+ long directTimeoutMillis) {
+ super(maxHeapMemory, maxHeapQueueSize, heapTimeoutMillis, maxDirectMemory, maxDirectQueueSize,
+ directTimeoutMillis);
+ this.collectorRegistry = collectorRegistry;
+
+ AsyncSemaphore heapMemoryLimiter = getLimiter(LimitType.HEAP_MEMORY);
+ AsyncSemaphore directMemoryLimiter = getLimiter(LimitType.DIRECT_MEMORY);
+
+ this.heapMemoryUsedBytes = register(Gauge.build(prometheusPrefix + "topic_list_heap_memory_used_bytes",
+ "Current heap memory used by topic listings")
+ .create()
+ .setChild(new Gauge.Child() {
+ @Override
+ public double get() {
+ return heapMemoryLimiter.getAcquiredPermits();
+ }
+ }));
+ this.otelHeapMemoryUsedGauge = openTelemetryMeter.gaugeBuilder("topic.list.heap.memory.used")
+ .setUnit("By")
+ .setDescription("Current heap memory used by topic listings")
+ .buildWithCallback(observableDoubleMeasurement -> {
+ observableDoubleMeasurement.record(heapMemoryLimiter.getAcquiredPermits());
+ });
+
+ this.heapMemoryLimitBytes = register(Gauge.build(prometheusPrefix + "topic_list_heap_memory_limit_bytes",
+ "Configured heap memory limit")
+ .create());
+ this.heapMemoryLimitBytes.set(maxHeapMemory);
+ this.otelHeapMemoryLimitGauge = openTelemetryMeter.gaugeBuilder("topic.list.heap.memory.limit")
+ .setUnit("By")
+ .setDescription("Configured heap memory limit")
+ .build();
+ this.otelHeapMemoryLimitGauge.set(maxHeapMemory);
+
+ this.directMemoryUsedBytes = register(Gauge.build(prometheusPrefix + "topic_list_direct_memory_used_bytes",
+ "Current direct memory used by topic listings")
+ .create()
+ .setChild(new Gauge.Child() {
+ @Override
+ public double get() {
+ return directMemoryLimiter.getAcquiredPermits();
+ }
+ }));
+ this.otelDirectMemoryUsedGauge = openTelemetryMeter.gaugeBuilder("topic.list.direct.memory.used")
+ .setUnit("By")
+ .setDescription("Current direct memory used by topic listings")
+ .buildWithCallback(observableDoubleMeasurement -> {
+ observableDoubleMeasurement.record(directMemoryLimiter.getAcquiredPermits());
+ });
+
+ this.directMemoryLimitBytes = register(Gauge.build(prometheusPrefix + "topic_list_direct_memory_limit_bytes",
+ "Configured direct memory limit")
+ .create());
+ this.directMemoryLimitBytes.set(maxDirectMemory);
+ this.otelDirectMemoryLimitGauge = openTelemetryMeter.gaugeBuilder("topic.list.direct.memory.limit")
+ .setUnit("By")
+ .setDescription("Configured direct memory limit")
+ .build();
+ this.otelDirectMemoryLimitGauge.set(maxDirectMemory);
+
+ this.heapQueueSize = register(Gauge.build(prometheusPrefix + "topic_list_heap_queue_size",
+ "Current heap memory limiter queue size")
+ .create()
+ .setChild(new Gauge.Child() {
+ @Override
+ public double get() {
+ return heapMemoryLimiter.getQueueSize();
+ }
+ }));
+ this.otelHeapQueueSize = openTelemetryMeter
+ .upDownCounterBuilder("topic.list.heap.queue.size")
+ .setDescription("Current heap memory limiter queue size")
+ .setUnit("1")
+ .buildWithCallback(observableLongMeasurement -> {
+ observableLongMeasurement.record(heapMemoryLimiter.getQueueSize());
+ });
+
+ this.heapQueueMaxSize = register(Gauge.build(prometheusPrefix + "topic_list_heap_queue_max_size",
+ "Maximum heap memory limiter queue size")
+ .create());
+ this.heapQueueMaxSize.set(maxHeapQueueSize);
+ LongGauge otelHeapQueueMaxSize = openTelemetryMeter
+ .gaugeBuilder("topic.list.heap.queue.max.size")
+ .setDescription("Maximum heap memory limiter queue size")
+ .setUnit("1")
+ .ofLongs()
+ .build();
+ otelHeapQueueMaxSize.set(maxHeapQueueSize);
Review Comment:
The `otelHeapQueueMaxSize` gauge is created as a local variable and cannot be closed in the `close()` method, potentially causing a resource leak.
##########
pulsar-common/src/main/java/org/apache/pulsar/common/semaphore/AsyncSemaphoreImpl.java:
##########
@@ -0,0 +1,288 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.common.semaphore;
+
+import io.netty.util.concurrent.DefaultThreadFactory;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicLongFieldUpdater;
+import java.util.function.BooleanSupplier;
+import java.util.function.LongConsumer;
+import org.apache.pulsar.common.util.Runnables;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of AsyncSemaphore with timeout and queue size limits.
+ */
+public class AsyncSemaphoreImpl implements AsyncSemaphore, AutoCloseable {
+ private static final Logger log = LoggerFactory.getLogger(AsyncSemaphoreImpl.class);
+
+ private final AtomicLong availablePermits;
+ private final ConcurrentLinkedQueue<PendingRequest> queue = new ConcurrentLinkedQueue<>();
+ private final long maxPermits;
+ private final int maxQueueSize;
+ private final long timeoutMillis;
+ private final ScheduledExecutorService executor;
+ private final boolean shutdownExecutor;
+ private final LongConsumer queueLatencyRecorder;
+ private final AtomicBoolean closed = new AtomicBoolean(false);
+ private final Runnable processQueueRunnable = Runnables.catchingAndLoggingThrowables(this::internalProcessQueue);
+
+ public AsyncSemaphoreImpl(long maxPermits, int maxQueueSize, long timeoutMillis) {
+ this(maxPermits, maxQueueSize, timeoutMillis, createExecutor(), true, null);
+ }
+
+ public AsyncSemaphoreImpl(long maxPermits, int maxQueueSize, long timeoutMillis,
+ ScheduledExecutorService executor, LongConsumer queueLatencyRecorder) {
+ this(maxPermits, maxQueueSize, timeoutMillis, executor, false, queueLatencyRecorder);
+ }
+
+ AsyncSemaphoreImpl(long maxPermits, int maxQueueSize, long timeoutMillis, ScheduledExecutorService executor,
+ boolean shutdownExecutor, LongConsumer queueLatencyRecorder) {
+ this.availablePermits = new AtomicLong(maxPermits);
+ this.maxPermits = maxPermits;
+ this.maxQueueSize = maxQueueSize;
+ this.timeoutMillis = timeoutMillis;
+ this.executor = executor;
+ this.shutdownExecutor = shutdownExecutor;
+ this.queueLatencyRecorder = queueLatencyRecorder;
+ }
+
+ private static ScheduledExecutorService createExecutor() {
+ return Executors.newSingleThreadScheduledExecutor(
+ new DefaultThreadFactory("async-semaphore-executor"));
+ }
+
+ @Override
+ public CompletableFuture<AsyncSemaphorePermit> acquire(long permits, BooleanSupplier isCancelled) {
+ return internalAcquire(permits, permits, isCancelled);
+ }
+
+ private CompletableFuture<AsyncSemaphorePermit> internalAcquire(long permits, long acquirePermits,
+ BooleanSupplier isCancelled) {
+ if (permits < 0) {
+ throw new IllegalArgumentException("Invalid permits value: " + permits);
+ }
+
+ CompletableFuture<AsyncSemaphorePermit> future = new CompletableFuture<>();
+
+ if (closed.get()) {
+ future.completeExceptionally(new PermitAcquireAlreadyClosedException("Semaphore is closed"));
+ return future;
+ }
+
+ if (queue.size() >= maxQueueSize) {
Review Comment:
The size() method on `ConcurrentLinkedQueue` iterates through all elements, making it an `O(n)` operation.
Maintaining a separate AtomicInteger counter to track queue size in O(1) time?
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/TopicListService.java:
##########
@@ -288,8 +294,13 @@ public void deleteTopicListWatcher(Long watcherId) {
*/
public void sendTopicListUpdate(long watcherId, String topicsHash, List<String> deletedTopics,
List<String> newTopics) {
- connection.getCommandSender().sendWatchTopicListUpdate(watcherId, newTopics, deletedTopics, topicsHash);
+ connection.getCommandSender().sendWatchTopicListUpdate(watcherId, newTopics, deletedTopics, topicsHash,
+ t -> {
+ // TODO add retry with backoff
+ log.warn(
+ "[{}] Cannot acquire direct memory tokens for sending topic list update. State will be "
+ + "inconsistent on the client. {}",
+ connection.toString(), t.getMessage());
+ });
Review Comment:
Extracting the error handling into a reusable method.
```java
private void handleSendFailure(String operation, Throwable t, long watcherId) {
// TODO: add retry with backoff
log.warn("[{}] Cannot acquire direct memory tokens for sending topic list update. State will be "
+ "inconsistent on the client. {}",
connection.toString(), t.getMessage());
}
...
// Line 206
connection.getCommandSender().sendWatchTopicListSuccess(requestId, watcherId, hash, topicList,
t -> handleSendFailure("topic list success", t, watcherId));
...
// Line 297
connection.getCommandSender().sendWatchTopicListUpdate(watcherId, newTopics, deletedTopics, topicsHash,
t -> handleSendFailure("topic list update", t, watcherId));
...
```
##########
pip/pip-442.md:
##########
@@ -132,24 +134,48 @@ public interface AsyncSemaphore {
* Returned future completes when permits are available.
* It will complete exceptionally with AsyncSemaphorePermitAcquireTimeoutException on timeout
Review Comment:
typo: `AsyncSemaphore.PermitAcquireTimeoutException`
##########
pulsar-broker-common/src/main/java/org/apache/pulsar/broker/topiclistlimit/TopicListMemoryLimiter.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.topiclistlimit;
+
+import io.opentelemetry.api.metrics.DoubleGauge;
+import io.opentelemetry.api.metrics.DoubleHistogram;
+import io.opentelemetry.api.metrics.LongCounter;
+import io.opentelemetry.api.metrics.LongGauge;
+import io.opentelemetry.api.metrics.Meter;
+import io.opentelemetry.api.metrics.ObservableDoubleGauge;
+import io.opentelemetry.api.metrics.ObservableLongUpDownCounter;
+import io.prometheus.client.Collector;
+import io.prometheus.client.CollectorRegistry;
+import io.prometheus.client.Counter;
+import io.prometheus.client.Gauge;
+import io.prometheus.client.Summary;
+import java.util.concurrent.TimeUnit;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.common.semaphore.AsyncDualMemoryLimiterImpl;
+import org.apache.pulsar.common.semaphore.AsyncSemaphore;
+
+/**
+ * Topic list memory limiter that exposes both Prometheus metrics and OpenTelemetry metrics.
+ */
+@Slf4j
+public class TopicListMemoryLimiter extends AsyncDualMemoryLimiterImpl {
+ private final CollectorRegistry collectorRegistry;
+ private final Gauge heapMemoryUsedBytes;
+ private final Gauge heapMemoryLimitBytes;
+ private final Gauge directMemoryUsedBytes;
+ private final Gauge directMemoryLimitBytes;
+ private final Gauge heapQueueSize;
+ private final Gauge heapQueueMaxSize;
+ private final Gauge directQueueSize;
+ private final Gauge directQueueMaxSize;
+ private final Summary heapWaitTimeMs;
+ private final Summary directWaitTimeMs;
+ private final Counter heapTimeoutTotal;
+ private final Counter directTimeoutTotal;
+ private final ObservableDoubleGauge otelHeapMemoryUsedGauge;
+ private final DoubleGauge otelHeapMemoryLimitGauge;
+ private final ObservableDoubleGauge otelDirectMemoryUsedGauge;
+ private final DoubleGauge otelDirectMemoryLimitGauge;
+ private final ObservableLongUpDownCounter otelHeapQueueSize;
+ private final ObservableLongUpDownCounter otelDirectQueueSize;
+ private final DoubleHistogram otelHeapWaitTime;
+ private final DoubleHistogram otelDirectWaitTime;
+ private final LongCounter otelHeapTimeoutTotal;
+ private final LongCounter otelDirectTimeoutTotal;
+
+ public TopicListMemoryLimiter(CollectorRegistry collectorRegistry, String prometheusPrefix,
+ Meter openTelemetryMeter,
+ long maxHeapMemory, int maxHeapQueueSize,
+ long heapTimeoutMillis, long maxDirectMemory, int maxDirectQueueSize,
+ long directTimeoutMillis) {
+ super(maxHeapMemory, maxHeapQueueSize, heapTimeoutMillis, maxDirectMemory, maxDirectQueueSize,
+ directTimeoutMillis);
+ this.collectorRegistry = collectorRegistry;
+
+ AsyncSemaphore heapMemoryLimiter = getLimiter(LimitType.HEAP_MEMORY);
+ AsyncSemaphore directMemoryLimiter = getLimiter(LimitType.DIRECT_MEMORY);
+
+ this.heapMemoryUsedBytes = register(Gauge.build(prometheusPrefix + "topic_list_heap_memory_used_bytes",
+ "Current heap memory used by topic listings")
+ .create()
+ .setChild(new Gauge.Child() {
+ @Override
+ public double get() {
+ return heapMemoryLimiter.getAcquiredPermits();
+ }
+ }));
+ this.otelHeapMemoryUsedGauge = openTelemetryMeter.gaugeBuilder("topic.list.heap.memory.used")
+ .setUnit("By")
+ .setDescription("Current heap memory used by topic listings")
+ .buildWithCallback(observableDoubleMeasurement -> {
+ observableDoubleMeasurement.record(heapMemoryLimiter.getAcquiredPermits());
+ });
+
+ this.heapMemoryLimitBytes = register(Gauge.build(prometheusPrefix + "topic_list_heap_memory_limit_bytes",
+ "Configured heap memory limit")
+ .create());
+ this.heapMemoryLimitBytes.set(maxHeapMemory);
+ this.otelHeapMemoryLimitGauge = openTelemetryMeter.gaugeBuilder("topic.list.heap.memory.limit")
+ .setUnit("By")
+ .setDescription("Configured heap memory limit")
+ .build();
+ this.otelHeapMemoryLimitGauge.set(maxHeapMemory);
+
+ this.directMemoryUsedBytes = register(Gauge.build(prometheusPrefix + "topic_list_direct_memory_used_bytes",
+ "Current direct memory used by topic listings")
+ .create()
+ .setChild(new Gauge.Child() {
+ @Override
+ public double get() {
+ return directMemoryLimiter.getAcquiredPermits();
+ }
+ }));
+ this.otelDirectMemoryUsedGauge = openTelemetryMeter.gaugeBuilder("topic.list.direct.memory.used")
+ .setUnit("By")
+ .setDescription("Current direct memory used by topic listings")
+ .buildWithCallback(observableDoubleMeasurement -> {
+ observableDoubleMeasurement.record(directMemoryLimiter.getAcquiredPermits());
+ });
+
+ this.directMemoryLimitBytes = register(Gauge.build(prometheusPrefix + "topic_list_direct_memory_limit_bytes",
+ "Configured direct memory limit")
+ .create());
+ this.directMemoryLimitBytes.set(maxDirectMemory);
+ this.otelDirectMemoryLimitGauge = openTelemetryMeter.gaugeBuilder("topic.list.direct.memory.limit")
+ .setUnit("By")
+ .setDescription("Configured direct memory limit")
+ .build();
+ this.otelDirectMemoryLimitGauge.set(maxDirectMemory);
+
+ this.heapQueueSize = register(Gauge.build(prometheusPrefix + "topic_list_heap_queue_size",
+ "Current heap memory limiter queue size")
+ .create()
+ .setChild(new Gauge.Child() {
+ @Override
+ public double get() {
+ return heapMemoryLimiter.getQueueSize();
+ }
+ }));
+ this.otelHeapQueueSize = openTelemetryMeter
+ .upDownCounterBuilder("topic.list.heap.queue.size")
+ .setDescription("Current heap memory limiter queue size")
+ .setUnit("1")
+ .buildWithCallback(observableLongMeasurement -> {
+ observableLongMeasurement.record(heapMemoryLimiter.getQueueSize());
+ });
+
+ this.heapQueueMaxSize = register(Gauge.build(prometheusPrefix + "topic_list_heap_queue_max_size",
+ "Maximum heap memory limiter queue size")
+ .create());
+ this.heapQueueMaxSize.set(maxHeapQueueSize);
+ LongGauge otelHeapQueueMaxSize = openTelemetryMeter
+ .gaugeBuilder("topic.list.heap.queue.max.size")
+ .setDescription("Maximum heap memory limiter queue size")
+ .setUnit("1")
+ .ofLongs()
+ .build();
+ otelHeapQueueMaxSize.set(maxHeapQueueSize);
+
+ this.directQueueSize = register(Gauge.build(prometheusPrefix + "topic_list_direct_queue_size",
+ "Current direct memory limiter queue size")
+ .create()
+ .setChild(new Gauge.Child() {
+ @Override
+ public double get() {
+ return directMemoryLimiter.getQueueSize();
+ }
+ }));
+ this.otelDirectQueueSize = openTelemetryMeter
+ .upDownCounterBuilder("topic.list.direct.queue.size")
+ .setDescription("Current direct memory limiter queue size")
+ .setUnit("1")
+ .buildWithCallback(observableLongMeasurement -> {
+ observableLongMeasurement.record(directMemoryLimiter.getQueueSize());
+ });
+
+ this.directQueueMaxSize = register(Gauge.build(prometheusPrefix + "topic_list_direct_queue_max_size",
+ "Maximum direct memory limiter queue size")
+ .create());
+ this.directQueueMaxSize.set(maxDirectQueueSize);
+ LongGauge otelDirectQueueMaxSize = openTelemetryMeter
+ .gaugeBuilder("topic.list.direct.queue.max.size")
+ .setDescription("Maximum direct memory limiter queue size")
+ .setUnit("1")
+ .ofLongs()
+ .build();
+ otelDirectQueueMaxSize.set(maxDirectQueueSize);
Review Comment:
The `otelDirectQueueMaxSize` gauge is created as a local variable and cannot be closed in the `close()` method, potentially causing a resource leak.
##########
pip/pip-442.md:
##########
@@ -132,24 +134,48 @@ public interface AsyncSemaphore {
* Returned future completes when permits are available.
* It will complete exceptionally with AsyncSemaphorePermitAcquireTimeoutException on timeout
* and exceptionally with AsyncSemaphorePermitAcquireQueueFullException when queue full
+ *
+ * @param permits number of permits to acquire
+ * @param isCancelled supplier that returns true if acquisition should be cancelled
* @return CompletableFuture that completes with permit when available
*/
- CompletableFuture<AsyncSemaphorePermitResult> acquire(long permits);
+ CompletableFuture<AsyncSemaphorePermit> acquire(long permits, BooleanSupplier isCancelled);
/**
* Acquire or release permits for previously acquired permits by updating the permits.
* Returns a future that completes when permits are available.
* It will complete exceptionally with AsyncSemaphorePermitAcquireTimeoutException on timeout
Review Comment:
typo: `AsyncSemaphore.PermitAcquireTimeoutException`
##########
pulsar-common/src/main/java/org/apache/pulsar/common/semaphore/AsyncSemaphore.java:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.common.semaphore;
+
+import java.util.concurrent.CompletableFuture;
+import java.util.function.BooleanSupplier;
+
+/**
+ * An abstraction for a generic asynchronous semaphore.
+ */
+public interface AsyncSemaphore {
+ /**
+ * Acquire permits from the semaphore.
+ * Returned future completes when permits are available.
+ * It will complete exceptionally with AsyncSemaphorePermitAcquireTimeoutException on timeout
+ * and exceptionally with AsyncSemaphorePermitAcquireQueueFullException when queue full
Review Comment:
typo: `AsyncSemaphore.PermitAcquireQueueFullException`
##########
pulsar-common/src/main/java/org/apache/pulsar/common/semaphore/AsyncSemaphore.java:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.common.semaphore;
+
+import java.util.concurrent.CompletableFuture;
+import java.util.function.BooleanSupplier;
+
+/**
+ * An abstraction for a generic asynchronous semaphore.
+ */
+public interface AsyncSemaphore {
+ /**
+ * Acquire permits from the semaphore.
+ * Returned future completes when permits are available.
+ * It will complete exceptionally with AsyncSemaphorePermitAcquireTimeoutException on timeout
+ * and exceptionally with AsyncSemaphorePermitAcquireQueueFullException when queue full
+ *
+ * @param permits number of permits to acquire
+ * @param isCancelled supplier that returns true if acquisition should be cancelled
+ * @return CompletableFuture that completes with permit when available
+ */
+ CompletableFuture<AsyncSemaphorePermit> acquire(long permits, BooleanSupplier isCancelled);
+
+ /**
+ * Acquire or release permits for previously acquired permits by updating the permits.
+ * Returns a future that completes when permits are available.
+ * It will complete exceptionally with AsyncSemaphorePermitAcquireTimeoutException on timeout
Review Comment:
typo: `AsyncSemaphore.PermitAcquireTimeoutException`
##########
pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/LookupProxyHandler.java:
##########
@@ -357,6 +357,59 @@ private void performGetTopicsOfNamespace(long clientRequestId,
});
}
+ private void internalPerformGetTopicsOfNamespace(long clientRequestId, String namespaceName, ClientCnx clientCnx,
+ ByteBuf command, long requestId) {
+ BooleanSupplier isPermitRequestCancelled = () -> !proxyConnection.ctx().channel().isActive();
+ maxTopicListInFlightLimiter.withAcquiredPermits(INITIAL_TOPIC_LIST_HEAP_PERMITS_SIZE,
+ AsyncDualMemoryLimiter.LimitType.HEAP_MEMORY, isPermitRequestCancelled, initialPermits -> {
+ return clientCnx.newGetTopicsOfNamespace(command, requestId).whenComplete((r, t) -> {
+ if (t != null) {
+ log.warn("[{}] Failed to get TopicsOfNamespace {}: {}", clientAddress, namespaceName,
+ t.getMessage());
+ writeAndFlush(Commands.newError(clientRequestId, getServerError(t), t.getMessage()));
+ } else {
+ long actualSize =
+ r.getNonPartitionedOrPartitionTopics().stream().mapToInt(String::length).sum();
+ maxTopicListInFlightLimiter.withUpdatedPermits(initialPermits, actualSize,
+ isPermitRequestCancelled, permits -> {
+ return handleWritingGetTopicsResponse(clientRequestId, r, isPermitRequestCancelled);
+ }, t2 -> {
+ log.warn("[{}] Failed to acquire actual heap memory permits for "
+ + "GetTopicsOfNamespace: {}", clientAddress, t2.getMessage());
+ writeAndFlush(Commands.newError(clientRequestId, ServerError.TooManyRequests,
+ "Failed due to heap memory limit exceeded"));
+
+ return CompletableFuture.completedFuture(null);
+ });
+ }
+ }).thenApply(__ -> null);
+ }, t -> {
+ log.warn("[{}] Failed to acquire initial heap memory permits for GetTopicsOfNamespace: {}",
+ clientAddress, t.getMessage());
+ writeAndFlush(Commands.newError(clientRequestId, ServerError.TooManyRequests,
+ "Failed due to heap memory limit exceeded"));
+
+ return CompletableFuture.completedFuture(null);
+ }).exceptionally(ex -> {
+ writeAndFlush(Commands.newError(clientRequestId, getServerError(ex), ex.getMessage()));
+ return null;
+ });
+ }
+
+ private CompletableFuture<Void> handleWritingGetTopicsResponse(long clientRequestId, GetTopicsResult r,
+ BooleanSupplier isCancelled) {
+ BaseCommand responseCommand = Commands.newGetTopicsOfNamespaceResponseCommand(
+ r.getNonPartitionedOrPartitionTopics(), r.getTopicsHash(), r.isFiltered(),
+ r.isChanged(), clientRequestId);
+ return acquireDirectMemoryPermitsAndWriteAndFlush(proxyConnection.ctx(), maxTopicListInFlightLimiter,
+ isCancelled, responseCommand, t -> {
+ log.warn("[{}] Failed to acquire actual direct memory permits for GetTopicsOfNamespace: {}",
+ clientAddress, t.getMessage());
+ writeAndFlush(Commands.newError(clientRequestId, ServerError.TooManyRequests,
+ "Failed due to heap memory limit exceeded"));
Review Comment:
"Failed due to direct memory limit exceeded"));
##########
pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/LookupProxyHandler.java:
##########
@@ -357,6 +357,59 @@ private void performGetTopicsOfNamespace(long clientRequestId,
});
}
+ private void internalPerformGetTopicsOfNamespace(long clientRequestId, String namespaceName, ClientCnx clientCnx,
+ ByteBuf command, long requestId) {
+ BooleanSupplier isPermitRequestCancelled = () -> !proxyConnection.ctx().channel().isActive();
+ maxTopicListInFlightLimiter.withAcquiredPermits(INITIAL_TOPIC_LIST_HEAP_PERMITS_SIZE,
+ AsyncDualMemoryLimiter.LimitType.HEAP_MEMORY, isPermitRequestCancelled, initialPermits -> {
+ return clientCnx.newGetTopicsOfNamespace(command, requestId).whenComplete((r, t) -> {
+ if (t != null) {
+ log.warn("[{}] Failed to get TopicsOfNamespace {}: {}", clientAddress, namespaceName,
+ t.getMessage());
+ writeAndFlush(Commands.newError(clientRequestId, getServerError(t), t.getMessage()));
+ } else {
+ long actualSize =
+ r.getNonPartitionedOrPartitionTopics().stream().mapToInt(String::length).sum();
Review Comment:
Java strings are UTF-16, so 2 bytes per character?
##########
pip/pip-442.md:
##########
@@ -132,24 +134,48 @@ public interface AsyncSemaphore {
* Returned future completes when permits are available.
* It will complete exceptionally with AsyncSemaphorePermitAcquireTimeoutException on timeout
* and exceptionally with AsyncSemaphorePermitAcquireQueueFullException when queue full
Review Comment:
typo: `AsyncSemaphore.PermitAcquireQueueFullException`
##########
pip/pip-442.md:
##########
@@ -163,227 +189,409 @@ public interface AsyncDualMemoryLimiter {
HEAP_MEMORY, // For heap memory allocation
DIRECT_MEMORY // For direct memory allocation
}
-
+
/**
* Acquire permits for the specified memory size.
* Returned future completes when memory permits are available.
- * It will complete exceptionally with AsyncSemaphorePermitAcquireTimeoutException on timeout
- * and exceptionally with AsyncSemaphorePermitAcquireQueueFullException when queue full
+ * It will complete exceptionally with AsyncSemaphore.PermitAcquireTimeoutException on timeout
+ * and exceptionally with AsyncSemaphore.PermitAcquireQueueFullException when queue full
+ *
+ * @param memorySize the size of memory to acquire permits for
+ * @param limitType the type of memory limit (HEAP_MEMORY or DIRECT_MEMORY)
+ * @param isCancelled supplier that returns true if acquisition should be cancelled
* @return CompletableFuture that completes with permit when available
*/
- CompletableFuture<AsyncDualMemoryLimiterPermit> acquire(long memorySize, LimitType limitType);
+ CompletableFuture<AsyncDualMemoryLimiterPermit> acquire(long memorySize, LimitType limitType,
+ BooleanSupplier isCancelled);
/**
* Acquire or release permits for previously acquired permits by updating the requested memory size.
* Returns a future that completes when permits are available.
- * It will complete exceptionally with AsyncSemaphorePermitAcquireTimeoutException on timeout
- * and exceptionally with AsyncSemaphorePermitAcquireQueueFullException when queue full
+ * It will complete exceptionally with AsyncSemaphore.PermitAcquireTimeoutException on timeout
+ * and exceptionally with AsyncSemaphore.PermitAcquireQueueFullException when queue full
+ * The provided permit is released when the permits are successfully acquired and the returned updated
+ * permit replaces the old instance.
+ *
+ * @param permit the previously acquired permit to update
+ * @param newMemorySize the new memory size to update to
+ * @param isCancelled supplier that returns true if update should be cancelled
* @return CompletableFuture that completes with permit when available
*/
- CompletableFuture<AsyncDualMemoryLimiterPermit> update(AsyncDualMemoryLimiterPermit permit, long newMemorySize);
-
+ CompletableFuture<AsyncDualMemoryLimiterPermit> update(AsyncDualMemoryLimiterPermit permit, long newMemorySize,
+ BooleanSupplier isCancelled);
+
/**
* Release previously acquired permit.
* Must be called to prevent memory permit leaks.
+ *
+ * @param permit the permit to release
*/
void release(AsyncDualMemoryLimiterPermit permit);
+ /**
+ * Execute the specified function with acquired permits and release the permits after the returned future completes.
+ * @param memorySize memory size to acquire permits for
+ * @param limitType memory limit type to acquire permits for
+ * @param function function to execute with acquired permits
+ * @return result of the function
+ * @param <T> type of the CompletableFuture returned by the function
+ */
+ default <T> CompletableFuture<T> withAcquiredPermits(long memorySize, LimitType limitType,
+ BooleanSupplier isCancelled,
+ Function<AsyncDualMemoryLimiterPermit,
+ CompletableFuture<T>> function,
+ Function<Throwable, CompletableFuture<T>>
+ permitAcquireErrorHandler) {
+ return AsyncDualMemoryLimiterUtil.withPermitsFuture(acquire(memorySize, limitType, isCancelled), function,
+ permitAcquireErrorHandler, this::release);
+ }
+
+ /**
+ * Executed the specified function with updated permits and release the permits after the returned future completes.
+ * @param initialPermit initial permit to update
+ * @param newMemorySize new memory size to update to
+ * @param function function to execute with updated permits
+ * @return result of the function
+ * @param <T> type of the CompletableFuture returned by the function
+ */
+ default <T> CompletableFuture<T> withUpdatedPermits(AsyncDualMemoryLimiterPermit initialPermit, long newMemorySize,
+ BooleanSupplier isCancelled,
+ Function<AsyncDualMemoryLimiterPermit,
+ CompletableFuture<T>> function,
+ Function<Throwable, CompletableFuture<T>>
+ permitAcquireErrorHandler) {
+ return AsyncDualMemoryLimiterUtil.withPermitsFuture(update(initialPermit, newMemorySize, isCancelled), function,
+ permitAcquireErrorHandler, this::release);
+ }
+}
+```
+
+#### AsyncDualMemoryLimiterUtil Helper
+
+A utility class provides helper methods for common patterns:
+
+```java
+public class AsyncDualMemoryLimiterUtil {
+ /**
+ * Execute a function with acquired permits and ensure permits are released after completion.
+ * This method handles the lifecycle of permits - acquisition, usage, and release, including error cases.
+ *
+ * @param permitsFuture Future that will complete with the required permits
+ * @param function Function to execute once permits are acquired that returns a CompletableFuture
+ * @param permitAcquireErrorHandler Handler for permit acquisition errors that returns a CompletableFuture
+ * @param releaser Consumer that handles releasing the permits
+ * @param <T> The type of result returned by the function
+ * @return CompletableFuture that completes with the result of the function execution
+ */
+ public static <T> CompletableFuture<T> withPermitsFuture(
+ CompletableFuture<AsyncDualMemoryLimiterPermit>
+ permitsFuture,
+ Function<AsyncDualMemoryLimiterPermit,
+ CompletableFuture<T>> function,
+ Function<Throwable, CompletableFuture<T>>
+ permitAcquireErrorHandler,
+ Consumer<AsyncDualMemoryLimiterPermit> releaser) {
+ // implementation omitted from PIP document
+ }
+
+ /**
+ * Acquires permits and writes the command as a response to the channel.
+ * Releases the permits after the response has been written to the socket or if the write fails.
+ *
+ * @param ctx the channel handler context used for writing the response
+ * @param dualMemoryLimiter the memory limiter used to acquire and release memory permits
+ * @param isCancelled supplier that indicates if the permit acquisition should be cancelled
+ * @param command the base command to serialize and write to the channel
+ * @param permitAcquireErrorHandler handler for errors that occur during permit acquisition
+ * @return a future that completes when the command has been written to the channel's outbound buffer
+ */
+ public static CompletableFuture<Void> acquireDirectMemoryPermitsAndWriteAndFlush(ChannelHandlerContext ctx,
+ AsyncDualMemoryLimiter
+ dualMemoryLimiter,
+ BooleanSupplier isCancelled,
+ BaseCommand command,
+ Consumer<Throwable>
+ permitAcquireErrorHandler
+ ) {
+ // implementation omitted from PIP document
+ }
}
```
#### Integration Points
-**1. Heap Memory Limiting (Post-Retrieval)**
+**1. Heap Memory Limiting (Post-Retrieval) - Broker**
-In `ServerCnx.handleGetTopicsOfNamespace`:
+In `ServerCnx.handleGetTopicsOfNamespace`, the implementation uses the helper methods:
```java
-// Acquire a fixed amount of permits initially since it's not known how much memory will be used
-// This will ensure that the operation continues only after it has the initial permits
-// It would be possible to use statistics for initial estimate, but this is simpler and sufficient
-maxTopicListInFlightLimiter.acquire(INITIAL_SIZE, AsyncDualMemoryLimiter.LimitType.HEAP_MEMORY)
- .thenCompose(initialPermit -> {
- getBrokerService().pulsar().getNamespaceService().getListOfUserTopics(namespaceName, mode)
- .thenCompose(topics -> {
- // Estimate memory after retrieval and update the permits to reflect the actual size
- long estimatedSize = topics.stream().mapToInt(String::length).sum();
- return maxTopicListInFlightLimiter
- .update(initialPermit, estimatedSize)
- .thenApply(permit -> Pair.of(topics, permit));
- })
- .thenAccept(topicsAndPermit -> {
- try {
- // Process and send response
- ...
- } finally {
- maxTopicListInFlightLimiter.release(topicsAndPermit.getRight());
- }
+ private void internalHandleGetTopicsOfNamespace(String namespace, NamespaceName namespaceName, long requestId,
+ CommandGetTopicsOfNamespace.Mode mode,
+ Optional<String> topicsPattern, Optional<String> topicsHash,
+ Semaphore lookupSemaphore) {
+ BooleanSupplier isPermitRequestCancelled = () -> !ctx().channel().isActive();
+ maxTopicListInFlightLimiter.withAcquiredPermits(INITIAL_TOPIC_LIST_HEAP_PERMITS_SIZE,
+ AsyncDualMemoryLimiter.LimitType.HEAP_MEMORY, isPermitRequestCancelled, initialPermits -> {
+ return getBrokerService().pulsar().getNamespaceService().getListOfUserTopics(namespaceName, mode)
+ .thenAccept(topics -> {
+ long actualSize = topics.stream().mapToInt(String::length).sum();
+ maxTopicListInFlightLimiter.withUpdatedPermits(initialPermits, actualSize,
+ isPermitRequestCancelled, permits -> {
+ boolean filterTopics = false;
+ // filter system topic
+ List<String> filteredTopics = topics;
+
+ if (enableSubscriptionPatternEvaluation && topicsPattern.isPresent()) {
+ if (topicsPattern.get().length() <= maxSubscriptionPatternLength) {
+ filterTopics = true;
+ filteredTopics = TopicList.filterTopics(filteredTopics, topicsPattern.get(),
+ topicsPatternImplementation);
+ } else {
+ log.info("[{}] Subscription pattern provided [{}] was longer than "
+ + "maximum {}.", remoteAddress, topicsPattern.get(),
+ maxSubscriptionPatternLength);
+ }
+ }
+ String hash = TopicList.calculateHash(filteredTopics);
+ boolean hashUnchanged = topicsHash.isPresent() && topicsHash.get().equals(hash);
+ if (hashUnchanged) {
+ filteredTopics = Collections.emptyList();
+ }
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Received CommandGetTopicsOfNamespace for namespace "
+ + "[//{}] by {}, size:{}", remoteAddress, namespace,
+ requestId,
+ topics.size());
+ }
+ commandSender.sendGetTopicsOfNamespaceResponse(filteredTopics, hash, filterTopics,
+ !hashUnchanged, requestId, ex -> {
+ log.warn("[{}] Failed to acquire direct memory permits for "
+ + "GetTopicsOfNamespace: {}", remoteAddress, ex.getMessage());
+ commandSender.sendErrorResponse(requestId, ServerError.TooManyRequests,
+ "Cannot acquire permits for direct memory");
+ });
+ return CompletableFuture.completedFuture(null);
+ }, t -> {
+ log.warn("[{}] Failed to acquire heap memory permits for "
+ + "GetTopicsOfNamespace: {}", remoteAddress, t.getMessage());
+ writeAndFlush(Commands.newError(requestId, ServerError.TooManyRequests,
+ "Failed due to heap memory limit exceeded"));
+ return CompletableFuture.completedFuture(null);
+ });
+ }).whenComplete((__, ___) -> {
+ lookupSemaphore.release();
+ }).exceptionally(ex -> {
+ log.warn("[{}] Error GetTopicsOfNamespace for namespace [//{}] by {}", remoteAddress,
+ namespace, requestId);
+ commandSender.sendErrorResponse(requestId,
+ BrokerServiceException.getClientErrorCode(new ServerMetadataException(ex)),
+ ex.getMessage());
+ return null;
+ });
+ }, t -> {
+ log.warn("[{}] Failed to acquire initial heap memory permits for GetTopicsOfNamespace: {}",
+ remoteAddress, t.getMessage());
+ writeAndFlush(Commands.newError(requestId, ServerError.TooManyRequests,
+ "Failed due to heap memory limit exceeded"));
+ lookupSemaphore.release();
+ return CompletableFuture.completedFuture(null);
});
- ...
- // For exceptional paths, initialPermit would need to be released
+}
```
-**2. Direct Memory Limiting (Pre-Serialization)**
+**2. Direct Memory Limiting (Pre-Serialization) - Broker**
-Modified `CommandSender` implementation:
+Modified `PulsarCommandSenderImpl`:
```java
@Override
public void sendGetTopicsOfNamespaceResponse(List<String> topics, String topicsHash,
- boolean filtered, boolean changed, long requestId) {
+ boolean filtered, boolean changed, long requestId,
+ Consumer<Throwable> permitAcquireErrorHandler) {
BaseCommand command = Commands.newGetTopicsOfNamespaceResponseCommand(topics, topicsHash,
filtered, changed, requestId);
safeIntercept(command, cnx);
- acquireMaxTopicListInFlightPermitsAndWriteAndFlush(command);
+ acquireDirectMemoryPermitsAndWriteAndFlush(cnx.ctx(), maxTopicListInFlightLimiter, () -> !cnx.isActive(),
+ command, permitAcquireErrorHandler);
}
+```
-private void acquireMaxTopicListInFlightPermitsAndWriteAndFlush(BaseCommand command) {
+The utility method implementation:
+
+```java
+public static CompletableFuture<Void> acquireDirectMemoryPermitsAndWriteAndFlush(ChannelHandlerContext ctx,
+ AsyncDualMemoryLimiter
+ dualMemoryLimiter,
+ BooleanSupplier isCancelled,
+ BaseCommand command,
+ Consumer<Throwable>
+ permitAcquireErrorHandler
+) {
// Calculate serialized size before acquiring permits
int serializedSize = command.getSerializedSize();
// Acquire permits
- maxTopicListInFlightLimiter.acquire(serializedSize, AsyncDualMemoryLimiter.LimitType.DIRECT_MEMORY)
- .thenAcceptAsync(permits -> {
+ return dualMemoryLimiter.acquire(serializedSize, AsyncDualMemoryLimiter.LimitType.DIRECT_MEMORY, isCancelled)
+ .whenComplete((permits, t) -> {
+ if (t != null) {
+ permitAcquireErrorHandler.accept(t);
+ return;
+ }
try {
// Serialize the response
ByteBuf outBuf = Commands.serializeWithPrecalculatedSerializedSize(command, serializedSize);
// Write the response
- cnx.ctx().writeAndFlush(outBuf).addListener(future -> {
+ ctx.writeAndFlush(outBuf).addListener(future -> {
// Release permits after the response has been written to the socket
- maxTopicListInFlightLimiter.release(permits);
+ dualMemoryLimiter.release(permits);
});
} catch (Exception e) {
Review Comment:
The `withPermitsFuture` method catches `Throwable (line 71)` to ensure permits are released even for Error subclasses.
```java
} catch (Throwable e) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "Denovo1998 (via GitHub)" <gi...@apache.org>.
Denovo1998 commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2426432010
##########
pulsar-common/src/main/java/org/apache/pulsar/common/semaphore/AsyncSemaphore.java:
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.common.semaphore;
+
+import java.util.concurrent.CompletableFuture;
+import java.util.function.BooleanSupplier;
+
+/**
+ * An abstraction for a generic asynchronous semaphore.
+ */
+public interface AsyncSemaphore {
+ /**
+ * Acquire permits from the semaphore.
+ * Returned future completes when permits are available.
+ * It will complete exceptionally with AsyncSemaphorePermitAcquireTimeoutException on timeout
+ * and exceptionally with AsyncSemaphorePermitAcquireQueueFullException when queue full
+ * @return CompletableFuture that completes with permit when available
+ */
+ CompletableFuture<AsyncSemaphorePermit> acquire(long permits, BooleanSupplier isCancelled);
+
+ /**
+ * Acquire or release permits for previously acquired permits by updating the permits.
+ * Returns a future that completes when permits are available.
+ * It will complete exceptionally with AsyncSemaphorePermitAcquireTimeoutException on timeout
+ * and exceptionally with AsyncSemaphorePermitAcquireQueueFullException when queue full
+ * @return CompletableFuture that completes with permit when available
+ */
Review Comment:
This should be synchronized with the modifications in the PIP documentation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2436951451
##########
pulsar-broker-common/src/main/java/org/apache/pulsar/broker/topiclistlimit/TopicListMemoryLimiter.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.topiclistlimit;
+
+import io.opentelemetry.api.metrics.DoubleGauge;
+import io.opentelemetry.api.metrics.DoubleHistogram;
+import io.opentelemetry.api.metrics.LongCounter;
+import io.opentelemetry.api.metrics.LongGauge;
+import io.opentelemetry.api.metrics.Meter;
+import io.opentelemetry.api.metrics.ObservableDoubleGauge;
+import io.opentelemetry.api.metrics.ObservableLongUpDownCounter;
+import io.prometheus.client.Collector;
+import io.prometheus.client.CollectorRegistry;
+import io.prometheus.client.Counter;
+import io.prometheus.client.Gauge;
+import io.prometheus.client.Summary;
+import java.util.concurrent.TimeUnit;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.common.semaphore.AsyncDualMemoryLimiterImpl;
+import org.apache.pulsar.common.semaphore.AsyncSemaphore;
+
+/**
+ * Topic list memory limiter that exposes both Prometheus metrics and OpenTelemetry metrics.
+ */
+@Slf4j
+public class TopicListMemoryLimiter extends AsyncDualMemoryLimiterImpl {
+ private final CollectorRegistry collectorRegistry;
+ private final Gauge heapMemoryUsedBytes;
+ private final Gauge heapMemoryLimitBytes;
+ private final Gauge directMemoryUsedBytes;
+ private final Gauge directMemoryLimitBytes;
+ private final Gauge heapQueueSize;
+ private final Gauge heapQueueMaxSize;
+ private final Gauge directQueueSize;
+ private final Gauge directQueueMaxSize;
+ private final Summary heapWaitTimeMs;
+ private final Summary directWaitTimeMs;
+ private final Counter heapTimeoutTotal;
+ private final Counter directTimeoutTotal;
+ private final ObservableDoubleGauge otelHeapMemoryUsedGauge;
+ private final DoubleGauge otelHeapMemoryLimitGauge;
+ private final ObservableDoubleGauge otelDirectMemoryUsedGauge;
+ private final DoubleGauge otelDirectMemoryLimitGauge;
+ private final ObservableLongUpDownCounter otelHeapQueueSize;
+ private final ObservableLongUpDownCounter otelDirectQueueSize;
+ private final DoubleHistogram otelHeapWaitTime;
+ private final DoubleHistogram otelDirectWaitTime;
+ private final LongCounter otelHeapTimeoutTotal;
+ private final LongCounter otelDirectTimeoutTotal;
+
+ public TopicListMemoryLimiter(CollectorRegistry collectorRegistry, String prometheusPrefix,
+ Meter openTelemetryMeter,
+ long maxHeapMemory, int maxHeapQueueSize,
+ long heapTimeoutMillis, long maxDirectMemory, int maxDirectQueueSize,
+ long directTimeoutMillis) {
+ super(maxHeapMemory, maxHeapQueueSize, heapTimeoutMillis, maxDirectMemory, maxDirectQueueSize,
+ directTimeoutMillis);
+ this.collectorRegistry = collectorRegistry;
+
+ AsyncSemaphore heapMemoryLimiter = getLimiter(LimitType.HEAP_MEMORY);
+ AsyncSemaphore directMemoryLimiter = getLimiter(LimitType.DIRECT_MEMORY);
+
+ this.heapMemoryUsedBytes = register(Gauge.build(prometheusPrefix + "topic_list_heap_memory_used_bytes",
+ "Current heap memory used by topic listings")
+ .create()
+ .setChild(new Gauge.Child() {
+ @Override
+ public double get() {
+ return heapMemoryLimiter.getAcquiredPermits();
+ }
+ }));
+ this.otelHeapMemoryUsedGauge = openTelemetryMeter.gaugeBuilder("topic.list.heap.memory.used")
+ .setUnit("By")
+ .setDescription("Current heap memory used by topic listings")
+ .buildWithCallback(observableDoubleMeasurement -> {
+ observableDoubleMeasurement.record(heapMemoryLimiter.getAcquiredPermits());
+ });
+
+ this.heapMemoryLimitBytes = register(Gauge.build(prometheusPrefix + "topic_list_heap_memory_limit_bytes",
+ "Configured heap memory limit")
+ .create());
+ this.heapMemoryLimitBytes.set(maxHeapMemory);
+ this.otelHeapMemoryLimitGauge = openTelemetryMeter.gaugeBuilder("topic.list.heap.memory.limit")
+ .setUnit("By")
+ .setDescription("Configured heap memory limit")
+ .build();
+ this.otelHeapMemoryLimitGauge.set(maxHeapMemory);
+
+ this.directMemoryUsedBytes = register(Gauge.build(prometheusPrefix + "topic_list_direct_memory_used_bytes",
+ "Current direct memory used by topic listings")
+ .create()
+ .setChild(new Gauge.Child() {
+ @Override
+ public double get() {
+ return directMemoryLimiter.getAcquiredPermits();
+ }
+ }));
+ this.otelDirectMemoryUsedGauge = openTelemetryMeter.gaugeBuilder("topic.list.direct.memory.used")
+ .setUnit("By")
+ .setDescription("Current direct memory used by topic listings")
+ .buildWithCallback(observableDoubleMeasurement -> {
+ observableDoubleMeasurement.record(directMemoryLimiter.getAcquiredPermits());
+ });
+
+ this.directMemoryLimitBytes = register(Gauge.build(prometheusPrefix + "topic_list_direct_memory_limit_bytes",
+ "Configured direct memory limit")
+ .create());
+ this.directMemoryLimitBytes.set(maxDirectMemory);
+ this.otelDirectMemoryLimitGauge = openTelemetryMeter.gaugeBuilder("topic.list.direct.memory.limit")
+ .setUnit("By")
+ .setDescription("Configured direct memory limit")
+ .build();
+ this.otelDirectMemoryLimitGauge.set(maxDirectMemory);
+
+ this.heapQueueSize = register(Gauge.build(prometheusPrefix + "topic_list_heap_queue_size",
+ "Current heap memory limiter queue size")
+ .create()
+ .setChild(new Gauge.Child() {
+ @Override
+ public double get() {
+ return heapMemoryLimiter.getQueueSize();
+ }
+ }));
+ this.otelHeapQueueSize = openTelemetryMeter
+ .upDownCounterBuilder("topic.list.heap.queue.size")
+ .setDescription("Current heap memory limiter queue size")
+ .setUnit("1")
+ .buildWithCallback(observableLongMeasurement -> {
+ observableLongMeasurement.record(heapMemoryLimiter.getQueueSize());
+ });
+
+ this.heapQueueMaxSize = register(Gauge.build(prometheusPrefix + "topic_list_heap_queue_max_size",
+ "Maximum heap memory limiter queue size")
+ .create());
+ this.heapQueueMaxSize.set(maxHeapQueueSize);
+ LongGauge otelHeapQueueMaxSize = openTelemetryMeter
+ .gaugeBuilder("topic.list.heap.queue.max.size")
+ .setDescription("Maximum heap memory limiter queue size")
+ .setUnit("1")
+ .ofLongs()
+ .build();
+ otelHeapQueueMaxSize.set(maxHeapQueueSize);
+
+ this.directQueueSize = register(Gauge.build(prometheusPrefix + "topic_list_direct_queue_size",
+ "Current direct memory limiter queue size")
+ .create()
+ .setChild(new Gauge.Child() {
+ @Override
+ public double get() {
+ return directMemoryLimiter.getQueueSize();
+ }
+ }));
+ this.otelDirectQueueSize = openTelemetryMeter
+ .upDownCounterBuilder("topic.list.direct.queue.size")
+ .setDescription("Current direct memory limiter queue size")
+ .setUnit("1")
+ .buildWithCallback(observableLongMeasurement -> {
+ observableLongMeasurement.record(directMemoryLimiter.getQueueSize());
+ });
+
+ this.directQueueMaxSize = register(Gauge.build(prometheusPrefix + "topic_list_direct_queue_max_size",
+ "Maximum direct memory limiter queue size")
+ .create());
+ this.directQueueMaxSize.set(maxDirectQueueSize);
+ LongGauge otelDirectQueueMaxSize = openTelemetryMeter
+ .gaugeBuilder("topic.list.direct.queue.max.size")
+ .setDescription("Maximum direct memory limiter queue size")
+ .setUnit("1")
+ .ofLongs()
+ .build();
+ otelDirectQueueMaxSize.set(maxDirectQueueSize);
+
+ this.heapWaitTimeMs = register(Summary.build(prometheusPrefix + "topic_list_heap_wait_time_ms",
Review Comment:
I assume you are referencing to ["Summary is widely used, but not aggregatable across brokers / labels"](https://github.com/apache/pulsar/blob/master/pip/pip-264.md#summary-is-widely-used-but-not-aggregatable-across-brokers--labels). In this case, it's a Prometheus metric and we have been using summaries. I think that there's only very few locations where Prometheus histograms are used in the Pulsar code base. IIRC, it's only for a few metadata store metrics.
Just wondering if we should just keep on using a summary in this case since there's not really a need to aggregate this metric.
For the Open Telemetry metric `topic.list.heap.wait.time`, `io.opentelemetry.api.metrics.DoubleHistogram` is already used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2466095497
##########
pulsar-broker-common/src/main/java/org/apache/pulsar/broker/topiclistlimit/TopicListSizeResultCache.java:
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.topiclistlimit;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReference;
+import org.apache.pulsar.common.api.proto.CommandGetTopicsOfNamespace;
+
+/**
+ * A cache for storing and managing topic list size estimates in namespaces.
+ * This class provides functionality to:
+ * - Cache and retrieve topic list size estimates for different namespaces and modes
+ * - Handle concurrent requests for topic list sizes efficiently
+ * - Maintain and update size estimates based on actual topic list sizes
+ * - Prevent thundering herd problems when multiple concurrent requests for a namespace are made without
+ * a previous size estimate
+ * The cache uses namespace name and topic list mode (PERSISTENT/NON_PERSISTENT/ALL) as keys
+ * and maintains size estimates that are refined with actual usage.
+ */
+public class TopicListSizeResultCache {
+ // 10kB initial estimate for topic list heap size
+ private static final long INITIAL_TOPIC_LIST_HEAP_SIZE = 10 * 1024;
+
+ private Cache<CacheKey, ResultHolder> topicListSizeCache = Caffeine.newBuilder()
+ .maximumSize(10000)
+ .build();
+
+ record CacheKey(String namespaceName, CommandGetTopicsOfNamespace.Mode mode) {
+ }
+
+ /**
+ * Holds the topic list size estimate and future for the topic list size.
+ * The size is returned by calling {@link #getSizeAsync()} method which is asynchronous.
+ * The estimate is updated by calling {@link #updateSize(long)} method.
+ */
+ public static class ResultHolder {
+ private final AtomicReference<CompletableFuture<Long>> topicListSizeFuture =
Review Comment:
This is necessary for the behavior that the first request passes through to resolve the size. This behavior is explained in the javadoc of the getSizeAsync method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2466122202
##########
pulsar-broker-common/src/main/java/org/apache/pulsar/broker/topiclistlimit/TopicListSizeResultCache.java:
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.topiclistlimit;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReference;
+import org.apache.pulsar.common.api.proto.CommandGetTopicsOfNamespace;
+
+/**
+ * A cache for storing and managing topic list size estimates in namespaces.
+ * This class provides functionality to:
+ * - Cache and retrieve topic list size estimates for different namespaces and modes
+ * - Handle concurrent requests for topic list sizes efficiently
+ * - Maintain and update size estimates based on actual topic list sizes
+ * - Prevent thundering herd problems when multiple concurrent requests for a namespace are made without
+ * a previous size estimate
+ * The cache uses namespace name and topic list mode (PERSISTENT/NON_PERSISTENT/ALL) as keys
+ * and maintains size estimates that are refined with actual usage.
+ */
+public class TopicListSizeResultCache {
+ // 10kB initial estimate for topic list heap size
+ private static final long INITIAL_TOPIC_LIST_HEAP_SIZE = 10 * 1024;
+
+ private Cache<CacheKey, ResultHolder> topicListSizeCache = Caffeine.newBuilder()
+ .maximumSize(10000)
+ .build();
+
+ record CacheKey(String namespaceName, CommandGetTopicsOfNamespace.Mode mode) {
+ }
+
+ /**
+ * Holds the topic list size estimate and future for the topic list size.
+ * The size is returned by calling {@link #getSizeAsync()} method which is asynchronous.
+ * The estimate is updated by calling {@link #updateSize(long)} method.
+ */
+ public static class ResultHolder {
+ private final AtomicReference<CompletableFuture<Long>> topicListSizeFuture =
+ new AtomicReference<>(null);
+ private final AtomicLong existingSizeRef = new AtomicLong(-1L);
+
+ /**
+ * Get the topic list size estimate. The first request will return the initial estimate
+ * and update the estimate based on the returned size of the topic list. Other concurrent requests
+ * will wait for the first request to complete and use the estimate of the first request.
+ * Subsequent requests will use estimate which gets updated based on the returned size of the topic list
+ * of each request.
+ * @return a future that will return the topic list size estimate
+ */
+ public CompletableFuture<Long> getSizeAsync() {
+ if (topicListSizeFuture.compareAndSet(null, new CompletableFuture<>())) {
+ // let the first request proceed with the initial estimate
+ return CompletableFuture.completedFuture(INITIAL_TOPIC_LIST_HEAP_SIZE);
+ } else {
+ // all other requests wait for the first one to complete
+ return topicListSizeFuture.get();
+ }
+ }
+
+ /**
+ * Update the topic list size estimate. The new estimated size will be updated by calculating the average
+ * of the existing and the new size. If the difference between the new and the existing size is less than 1,
+ * no update will be done.
+ * @param actualSize the actual size of the topic list
+ */
+ public void updateSize(long actualSize) {
Review Comment:
It's intentional that it's dropped, I guess adding a comment explaining the logic could be helpful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2440988137
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/ServerCnx.java:
##########
@@ -2610,6 +2581,99 @@ protected void handleGetTopicsOfNamespace(CommandGetTopicsOfNamespace commandGet
}
}
+ private void internalHandleGetTopicsOfNamespace(String namespace, NamespaceName namespaceName, long requestId,
+ CommandGetTopicsOfNamespace.Mode mode,
+ Optional<String> topicsPattern, Optional<String> topicsHash,
+ Semaphore lookupSemaphore) {
+ BooleanSupplier isPermitRequestCancelled = () -> !ctx().channel().isActive();
+ TopicListSizeResultCache.ResultHolder
+ listSizeHolder = service.getTopicListSizeResultCache().getTopicListSize(namespaceName.toString(), mode);
+ listSizeHolder.getSizeAsync().thenAccept(initialSize -> {
+ maxTopicListInFlightLimiter.withAcquiredPermits(initialSize,
+ AsyncDualMemoryLimiter.LimitType.HEAP_MEMORY, isPermitRequestCancelled, initialPermits -> {
+ return getBrokerService().pulsar().getNamespaceService()
+ .getListOfUserTopics(namespaceName, mode)
+ .thenAccept(topics -> {
+ long actualSize = topics.stream()
+ .mapToInt(ByteBufUtil::utf8Bytes) // convert character count to bytes
+ .map(n -> n + 32) // add 32 bytes overhead for each entry
Review Comment:
I also realized that java.lang.String contains several fields that would be consuming some amount of bytes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#issuecomment-3416786658
I noticed a separate heap memory issue related to topic listing operations: the childrenCache in AbstractMetadataStore. There's a separate PR to address the unbounded cache: https://github.com/apache/pulsar/pull/24868.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2479181621
##########
pulsar-common/src/main/java/org/apache/pulsar/common/semaphore/AsyncSemaphoreImpl.java:
##########
@@ -0,0 +1,330 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.common.semaphore;
+
+import io.netty.util.concurrent.DefaultThreadFactory;
+import java.util.Queue;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicLongFieldUpdater;
+import java.util.function.BooleanSupplier;
+import java.util.function.LongConsumer;
+import org.apache.pulsar.common.util.Runnables;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of AsyncSemaphore with timeout and queue size limits.
+ */
+public class AsyncSemaphoreImpl implements AsyncSemaphore, AutoCloseable {
+ private static final Logger log = LoggerFactory.getLogger(AsyncSemaphoreImpl.class);
+
+ private final AtomicLong availablePermits;
+ private final Queue<PendingRequest> queue;
+ private final long maxPermits;
+ private final long timeoutMillis;
+ private final ScheduledExecutorService executor;
+ private final boolean shutdownExecutor;
+ private final LongConsumer queueLatencyRecorder;
+ private final AtomicBoolean closed = new AtomicBoolean(false);
+ private final Runnable processQueueRunnable = Runnables.catchingAndLoggingThrowables(this::internalProcessQueue);
+
+ /**
+ * Creates an AsyncSemaphoreImpl with the given parameters.
+ * @param maxPermits max number of permits available for acquisition, set to <= 0 for unbounded semaphore (not
+ * recommended)
+ * @param maxQueueSize max number of requests that can be queued, set to <= 0 for unbounded queue (not recommended)
+ * @param timeoutMillis timeout in milliseconds for acquiring permits
+ */
+ public AsyncSemaphoreImpl(long maxPermits, int maxQueueSize, long timeoutMillis) {
+ this(maxPermits, maxQueueSize, timeoutMillis, maxPermits > 0 ? createExecutor() : null, maxPermits > 0, null);
+ }
+
+ /**
+ * Creates an AsyncSemaphoreImpl with the given parameters.
+ * @param maxPermits max number of permits available for acquisition, set to <= 0 for unbounded semaphore (not
+ * recommended)
+ * @param maxQueueSize max number of requests that can be queued, set to <= 0 for unbounded queue (not recommended)
+ * @param timeoutMillis timeout in milliseconds for acquiring permits
+ * @param executor executor service to use for scheduling timeouts, it is expected to be single threaded
+ * @param queueLatencyRecorder consumer to record queue latency, Long.MAX_VALUE is used for requests that timed out
+ */
+ public AsyncSemaphoreImpl(long maxPermits, int maxQueueSize, long timeoutMillis,
+ ScheduledExecutorService executor, LongConsumer queueLatencyRecorder) {
+ this(maxPermits, maxQueueSize, timeoutMillis, executor, false, queueLatencyRecorder);
+ }
+
+ AsyncSemaphoreImpl(long maxPermits, int maxQueueSize, long timeoutMillis, ScheduledExecutorService executor,
+ boolean shutdownExecutor, LongConsumer queueLatencyRecorder) {
+ this.availablePermits = new AtomicLong(maxPermits);
+ this.maxPermits = maxPermits;
+ this.queue = maxQueueSize > 0 ? new ArrayBlockingQueue<>(maxQueueSize) : new LinkedBlockingQueue<>();
+ this.timeoutMillis = timeoutMillis;
+ this.executor = executor;
+ this.shutdownExecutor = shutdownExecutor;
+ this.queueLatencyRecorder = queueLatencyRecorder;
+ }
+
+ private static ScheduledExecutorService createExecutor() {
+ return Executors.newSingleThreadScheduledExecutor(
+ new DefaultThreadFactory("async-semaphore-executor"));
+ }
+
+ @Override
+ public CompletableFuture<AsyncSemaphorePermit> acquire(long permits, BooleanSupplier isCancelled) {
+ return internalAcquire(permits, permits, isCancelled);
+ }
+
+ private CompletableFuture<AsyncSemaphorePermit> internalAcquire(long permits, long acquirePermits,
+ BooleanSupplier isCancelled) {
+ validatePermits(permits);
+
+ // if maximum permits is <= 0, then the semaphore is unbounded
+ if (isUnbounded()) {
+ return CompletableFuture.completedFuture(new SemaphorePermit(permits));
+ }
+
+ CompletableFuture<AsyncSemaphorePermit> future = new CompletableFuture<>();
+
+ if (closed.get()) {
+ future.completeExceptionally(new PermitAcquireAlreadyClosedException("Semaphore is closed"));
+ return future;
+ }
+
+ PendingRequest request = new PendingRequest(permits, acquirePermits, future, isCancelled);
+ if (!queue.offer(request)) {
+ future.completeExceptionally(new PermitAcquireQueueFullException(
+ "Semaphore queue is full"));
+ return future;
+ }
+ // Schedule timeout
+ ScheduledFuture<?> timeoutTask = executor.schedule(() -> {
+ if (!request.future.isDone() && queue.remove(request)) {
+ // timeout is recorded with Long.MAX_VALUE as the age
+ recordQueueLatency(Long.MAX_VALUE);
+ // also record the time in the queue
+ recordQueueLatency(request.getAgeNanos());
+ future.completeExceptionally(new PermitAcquireTimeoutException(
+ "Permit acquisition timed out"));
+ // the next request might have smaller permits and that might be processed
+ processQueue();
+ }
+ }, timeoutMillis, TimeUnit.MILLISECONDS);
+ request.setTimeoutTask(timeoutTask);
+
+ processQueue();
+ return future;
+ }
+
+ private void validatePermits(long permits) {
+ if (permits < 0) {
+ throw new IllegalArgumentException("Invalid negative permits value: " + permits);
+ }
+ if (!isUnbounded() && permits > maxPermits) {
+ throw new IllegalArgumentException(
+ "Requested permits=" + permits + " is larger than maxPermits=" + maxPermits);
+ }
+ }
+
+ private boolean isUnbounded() {
+ return maxPermits <= 0;
+ }
+
+ private void recordQueueLatency(long ageNanos) {
+ if (queueLatencyRecorder != null) {
+ queueLatencyRecorder.accept(ageNanos);
+ }
+ }
+
+ @Override
+ public CompletableFuture<AsyncSemaphorePermit> update(AsyncSemaphorePermit permit, long newPermits,
+ BooleanSupplier isCancelled) {
+ validatePermits(newPermits);
+ if (isUnbounded()) {
+ return CompletableFuture.completedFuture(new SemaphorePermit(newPermits));
+ }
+ long oldPermits = permit.getPermits();
+ long additionalPermits = newPermits - oldPermits;
+ // mark the old permits as released without adding the permits to availablePermits
+ castToImplementation(permit).releasePermits();
+ if (additionalPermits > 0) {
+ return internalAcquire(newPermits, additionalPermits, isCancelled);
+ } else {
+ // new permits are less than the old ones, so we return the difference
+ availablePermits.addAndGet(-additionalPermits);
+ processQueue();
+ // return the new permits immediately
+ return CompletableFuture.completedFuture(new SemaphorePermit(newPermits));
Review Comment:
I added test cases to cover your concerns. There's no bug.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "poorbarcode (via GitHub)" <gi...@apache.org>.
poorbarcode commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2476710868
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/ServerCnx.java:
##########
@@ -2610,6 +2581,93 @@ protected void handleGetTopicsOfNamespace(CommandGetTopicsOfNamespace commandGet
}
}
+ private void internalHandleGetTopicsOfNamespace(String namespace, NamespaceName namespaceName, long requestId,
+ CommandGetTopicsOfNamespace.Mode mode,
+ Optional<String> topicsPattern, Optional<String> topicsHash,
+ Semaphore lookupSemaphore) {
+ BooleanSupplier isPermitRequestCancelled = () -> !ctx().channel().isActive();
+ TopicListSizeResultCache.ResultHolder
+ listSizeHolder = service.getTopicListSizeResultCache().getTopicListSize(namespaceName.toString(), mode);
+ listSizeHolder.getSizeAsync().thenAccept(initialSize -> {
+ maxTopicListInFlightLimiter.withAcquiredPermits(initialSize,
+ AsyncDualMemoryLimiter.LimitType.HEAP_MEMORY, isPermitRequestCancelled, initialPermits -> {
+ return getBrokerService().pulsar().getNamespaceService()
+ .getListOfUserTopics(namespaceName, mode)
+ .thenAccept(topics -> {
+ long actualSize = TopicListMemoryLimiter.estimateTopicListSize(topics);
+ listSizeHolder.updateSize(actualSize);
+ maxTopicListInFlightLimiter.withUpdatedPermits(initialPermits, actualSize,
Review Comment:
No, the implementation is like follows, which never covered the case I said above
- update
- call `internalAcquire` if `realSize > estimatedSize`, see also https://github.com/apache/pulsar/pull/24833/files#diff-1daf74806e4d1f752d9576bd63d71678a7a46cbc7a1cf56f71ba88719f87d346R173-R175
- `internalAcquire` will pending the current thread or throws errors, see also https://github.com/apache/pulsar/pull/24833/files#diff-1daf74806e4d1f752d9576bd63d71678a7a46cbc7a1cf56f71ba88719f87d346R117-R122
- **method update**
```java
public CompletableFuture<AsyncSemaphorePermit> update(AsyncSemaphorePermit permit, long newPermits,
BooleanSupplier isCancelled) {
long oldPermits = permit.getPermits();
long additionalPermits = newPermits - oldPermits;
if (additionalPermits > 0) {
return internalAcquire(newPermits, additionalPermits, isCancelled);
}
}
```
- **pending requests**
```java
private CompletableFuture<AsyncSemaphorePermit> internalAcquire(long permits, long acquirePermits,
BooleanSupplier isCancelled) {
validatePermits(permits);
PendingRequest request = new PendingRequest(permits, acquirePermits, future, isCancelled);
if (!queue.offer(request)) {
future.completeExceptionally(new PermitAcquireQueueFullException(
"Semaphore queue is full"));
return future;
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2479088694
##########
pulsar-common/src/main/java/org/apache/pulsar/common/semaphore/AsyncSemaphoreImpl.java:
##########
@@ -0,0 +1,330 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.common.semaphore;
+
+import io.netty.util.concurrent.DefaultThreadFactory;
+import java.util.Queue;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicLongFieldUpdater;
+import java.util.function.BooleanSupplier;
+import java.util.function.LongConsumer;
+import org.apache.pulsar.common.util.Runnables;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of AsyncSemaphore with timeout and queue size limits.
+ */
+public class AsyncSemaphoreImpl implements AsyncSemaphore, AutoCloseable {
+ private static final Logger log = LoggerFactory.getLogger(AsyncSemaphoreImpl.class);
+
+ private final AtomicLong availablePermits;
+ private final Queue<PendingRequest> queue;
+ private final long maxPermits;
+ private final long timeoutMillis;
+ private final ScheduledExecutorService executor;
+ private final boolean shutdownExecutor;
+ private final LongConsumer queueLatencyRecorder;
+ private final AtomicBoolean closed = new AtomicBoolean(false);
+ private final Runnable processQueueRunnable = Runnables.catchingAndLoggingThrowables(this::internalProcessQueue);
+
+ /**
+ * Creates an AsyncSemaphoreImpl with the given parameters.
+ * @param maxPermits max number of permits available for acquisition, set to <= 0 for unbounded semaphore (not
+ * recommended)
+ * @param maxQueueSize max number of requests that can be queued, set to <= 0 for unbounded queue (not recommended)
+ * @param timeoutMillis timeout in milliseconds for acquiring permits
+ */
+ public AsyncSemaphoreImpl(long maxPermits, int maxQueueSize, long timeoutMillis) {
+ this(maxPermits, maxQueueSize, timeoutMillis, maxPermits > 0 ? createExecutor() : null, maxPermits > 0, null);
+ }
+
+ /**
+ * Creates an AsyncSemaphoreImpl with the given parameters.
+ * @param maxPermits max number of permits available for acquisition, set to <= 0 for unbounded semaphore (not
+ * recommended)
+ * @param maxQueueSize max number of requests that can be queued, set to <= 0 for unbounded queue (not recommended)
+ * @param timeoutMillis timeout in milliseconds for acquiring permits
+ * @param executor executor service to use for scheduling timeouts, it is expected to be single threaded
+ * @param queueLatencyRecorder consumer to record queue latency, Long.MAX_VALUE is used for requests that timed out
+ */
+ public AsyncSemaphoreImpl(long maxPermits, int maxQueueSize, long timeoutMillis,
+ ScheduledExecutorService executor, LongConsumer queueLatencyRecorder) {
+ this(maxPermits, maxQueueSize, timeoutMillis, executor, false, queueLatencyRecorder);
+ }
+
+ AsyncSemaphoreImpl(long maxPermits, int maxQueueSize, long timeoutMillis, ScheduledExecutorService executor,
+ boolean shutdownExecutor, LongConsumer queueLatencyRecorder) {
+ this.availablePermits = new AtomicLong(maxPermits);
+ this.maxPermits = maxPermits;
+ this.queue = maxQueueSize > 0 ? new ArrayBlockingQueue<>(maxQueueSize) : new LinkedBlockingQueue<>();
+ this.timeoutMillis = timeoutMillis;
+ this.executor = executor;
+ this.shutdownExecutor = shutdownExecutor;
+ this.queueLatencyRecorder = queueLatencyRecorder;
+ }
+
+ private static ScheduledExecutorService createExecutor() {
+ return Executors.newSingleThreadScheduledExecutor(
+ new DefaultThreadFactory("async-semaphore-executor"));
+ }
+
+ @Override
+ public CompletableFuture<AsyncSemaphorePermit> acquire(long permits, BooleanSupplier isCancelled) {
+ return internalAcquire(permits, permits, isCancelled);
+ }
+
+ private CompletableFuture<AsyncSemaphorePermit> internalAcquire(long permits, long acquirePermits,
+ BooleanSupplier isCancelled) {
+ validatePermits(permits);
+
+ // if maximum permits is <= 0, then the semaphore is unbounded
+ if (isUnbounded()) {
+ return CompletableFuture.completedFuture(new SemaphorePermit(permits));
+ }
+
+ CompletableFuture<AsyncSemaphorePermit> future = new CompletableFuture<>();
+
+ if (closed.get()) {
+ future.completeExceptionally(new PermitAcquireAlreadyClosedException("Semaphore is closed"));
+ return future;
+ }
+
+ PendingRequest request = new PendingRequest(permits, acquirePermits, future, isCancelled);
+ if (!queue.offer(request)) {
+ future.completeExceptionally(new PermitAcquireQueueFullException(
+ "Semaphore queue is full"));
+ return future;
+ }
+ // Schedule timeout
+ ScheduledFuture<?> timeoutTask = executor.schedule(() -> {
+ if (!request.future.isDone() && queue.remove(request)) {
+ // timeout is recorded with Long.MAX_VALUE as the age
+ recordQueueLatency(Long.MAX_VALUE);
+ // also record the time in the queue
+ recordQueueLatency(request.getAgeNanos());
+ future.completeExceptionally(new PermitAcquireTimeoutException(
+ "Permit acquisition timed out"));
+ // the next request might have smaller permits and that might be processed
+ processQueue();
+ }
+ }, timeoutMillis, TimeUnit.MILLISECONDS);
+ request.setTimeoutTask(timeoutTask);
+
+ processQueue();
+ return future;
+ }
+
+ private void validatePermits(long permits) {
+ if (permits < 0) {
+ throw new IllegalArgumentException("Invalid negative permits value: " + permits);
+ }
+ if (!isUnbounded() && permits > maxPermits) {
+ throw new IllegalArgumentException(
+ "Requested permits=" + permits + " is larger than maxPermits=" + maxPermits);
+ }
+ }
+
+ private boolean isUnbounded() {
+ return maxPermits <= 0;
+ }
+
+ private void recordQueueLatency(long ageNanos) {
+ if (queueLatencyRecorder != null) {
+ queueLatencyRecorder.accept(ageNanos);
+ }
+ }
+
+ @Override
+ public CompletableFuture<AsyncSemaphorePermit> update(AsyncSemaphorePermit permit, long newPermits,
+ BooleanSupplier isCancelled) {
+ validatePermits(newPermits);
+ if (isUnbounded()) {
+ return CompletableFuture.completedFuture(new SemaphorePermit(newPermits));
Review Comment:
The above link doesn't resolve anymore. In the unbounded case, the semaphore is like a no-op. If someone would like to turn off the semaphore behavior, it's possible by setting maxPermits to 0. That makes it just hand out the permits immediately. There's no bug in this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace [pulsar]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2506849231
##########
pulsar-common/src/main/java/org/apache/pulsar/common/semaphore/AsyncDualMemoryLimiter.java:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.common.semaphore;
+
+import java.util.concurrent.CompletableFuture;
+import java.util.function.BooleanSupplier;
+import java.util.function.Function;
+
+/**
+ * An abstraction for an asynchronous memory semaphore that tracks separate limits for heap and direct memory.
+ */
+public interface AsyncDualMemoryLimiter {
+ enum LimitType {
+ HEAP_MEMORY, // For heap memory allocation
+ DIRECT_MEMORY // For direct memory allocation
+ }
+
+ /**
+ * Acquire permits for the specified memory size.
+ * Returned future completes when memory permits are available.
+ * It will complete exceptionally with AsyncSemaphore.PermitAcquireTimeoutException on timeout
+ * and exceptionally with AsyncSemaphore.PermitAcquireQueueFullException when queue full
+ *
+ * @param memorySize the size of memory to acquire permits for
+ * @param limitType the type of memory limit (HEAP_MEMORY or DIRECT_MEMORY)
+ * @param isCancelled supplier that returns true if acquisition should be cancelled
+ * @return CompletableFuture that completes with permit when available
+ */
+ CompletableFuture<AsyncDualMemoryLimiterPermit> acquire(long memorySize, LimitType limitType,
+ BooleanSupplier isCancelled);
+
+ /**
+ * Acquire or release permits for previously acquired permits by updating the requested memory size.
+ * Returns a future that completes when permits are available.
+ * It will complete exceptionally with AsyncSemaphore.PermitAcquireTimeoutException on timeout
+ * and exceptionally with AsyncSemaphore.PermitAcquireQueueFullException when queue full
+ * The provided permit is released when the permits are successfully acquired and the returned updated
+ * permit replaces the old instance.
+ *
+ * @param permit the previously acquired permit to update
+ * @param newMemorySize the new memory size to update to
+ * @param isCancelled supplier that returns true if update should be cancelled
+ * @return CompletableFuture that completes with permit when available
+ */
+ CompletableFuture<AsyncDualMemoryLimiterPermit> update(AsyncDualMemoryLimiterPermit permit, long newMemorySize,
+ BooleanSupplier isCancelled);
+
+ /**
+ * Release previously acquired permit.
+ * Must be called to prevent memory permit leaks.
+ *
+ * @param permit the permit to release
+ */
+ void release(AsyncDualMemoryLimiterPermit permit);
+ /**
+ * Execute the specified function with acquired permits and release the permits after the returned future completes.
+ * @param memorySize memory size to acquire permits for
+ * @param limitType memory limit type to acquire permits for
+ * @param function function to execute with acquired permits
+ * @return result of the function
+ * @param <T> type of the CompletableFuture returned by the function
+ */
+ default <T> CompletableFuture<T> withAcquiredPermits(long memorySize, LimitType limitType,
+ BooleanSupplier isCancelled,
+ Function<AsyncDualMemoryLimiterPermit,
+ CompletableFuture<T>> function,
+ Function<Throwable, CompletableFuture<T>>
+ permitAcquireErrorHandler) {
+ return AsyncDualMemoryLimiterUtil.withPermitsFuture(acquire(memorySize, limitType, isCancelled), function,
+ permitAcquireErrorHandler, this::release);
+ }
+
+ /**
+ * Executed the specified function with updated permits and release the permits after the returned future completes.
Review Comment:
Corrected spelling of 'Executed' to 'Execute' to match the imperative verb form used in method 'withAcquiredPermits'.
```suggestion
* Execute the specified function with updated permits and release the permits after the returned future completes.
```
##########
pulsar-broker-common/src/main/java/org/apache/pulsar/broker/topiclistlimit/TopicListMemoryLimiter.java:
##########
@@ -0,0 +1,312 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.topiclistlimit;
+
+import io.netty.buffer.ByteBufUtil;
+import io.opentelemetry.api.metrics.DoubleGauge;
+import io.opentelemetry.api.metrics.DoubleHistogram;
+import io.opentelemetry.api.metrics.LongCounter;
+import io.opentelemetry.api.metrics.LongGauge;
+import io.opentelemetry.api.metrics.Meter;
+import io.opentelemetry.api.metrics.ObservableDoubleGauge;
+import io.opentelemetry.api.metrics.ObservableLongUpDownCounter;
+import io.prometheus.client.Collector;
+import io.prometheus.client.CollectorRegistry;
+import io.prometheus.client.Counter;
+import io.prometheus.client.Gauge;
+import io.prometheus.client.Summary;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.common.semaphore.AsyncDualMemoryLimiterImpl;
+import org.apache.pulsar.common.semaphore.AsyncSemaphore;
+
+/**
+ * Topic list memory limiter that exposes both Prometheus metrics and OpenTelemetry metrics.
+ */
+@Slf4j
+public class TopicListMemoryLimiter extends AsyncDualMemoryLimiterImpl {
+ private final CollectorRegistry collectorRegistry;
+ private final Gauge heapMemoryUsedBytes;
+ private final Gauge heapMemoryLimitBytes;
+ private final Gauge directMemoryUsedBytes;
+ private final Gauge directMemoryLimitBytes;
+ private final Gauge heapQueueSize;
+ private final Gauge heapQueueMaxSize;
+ private final Gauge directQueueSize;
+ private final Gauge directQueueMaxSize;
+ private final Summary heapWaitTimeMs;
+ private final Summary directWaitTimeMs;
+ private final Counter heapTimeoutTotal;
+ private final Counter directTimeoutTotal;
+ private final ObservableDoubleGauge otelHeapMemoryUsedGauge;
+ private final DoubleGauge otelHeapMemoryLimitGauge;
+ private final ObservableDoubleGauge otelDirectMemoryUsedGauge;
+ private final DoubleGauge otelDirectMemoryLimitGauge;
+ private final ObservableLongUpDownCounter otelHeapQueueSize;
+ private final ObservableLongUpDownCounter otelDirectQueueSize;
+ private final DoubleHistogram otelHeapWaitTime;
+ private final DoubleHistogram otelDirectWaitTime;
+ private final LongCounter otelHeapTimeoutTotal;
+ private final LongCounter otelDirectTimeoutTotal;
+
+ public TopicListMemoryLimiter(CollectorRegistry collectorRegistry, String prometheusPrefix,
+ Meter openTelemetryMeter,
+ long maxHeapMemory, int maxHeapQueueSize,
+ long heapTimeoutMillis, long maxDirectMemory, int maxDirectQueueSize,
+ long directTimeoutMillis) {
+ super(maxHeapMemory, maxHeapQueueSize, heapTimeoutMillis, maxDirectMemory, maxDirectQueueSize,
+ directTimeoutMillis);
+ this.collectorRegistry = collectorRegistry;
+
+ AsyncSemaphore heapMemoryLimiter = getLimiter(LimitType.HEAP_MEMORY);
+ AsyncSemaphore directMemoryLimiter = getLimiter(LimitType.DIRECT_MEMORY);
+
+ this.heapMemoryUsedBytes = register(Gauge.build(prometheusPrefix + "topic_list_heap_memory_used_bytes",
+ "Current heap memory used by topic listings")
+ .create()
+ .setChild(new Gauge.Child() {
+ @Override
+ public double get() {
+ return heapMemoryLimiter.getAcquiredPermits();
+ }
+ }));
+ this.otelHeapMemoryUsedGauge = openTelemetryMeter.gaugeBuilder("topic.list.heap.memory.used")
+ .setUnit("By")
+ .setDescription("Current heap memory used by topic listings")
+ .buildWithCallback(observableDoubleMeasurement -> {
+ observableDoubleMeasurement.record(heapMemoryLimiter.getAcquiredPermits());
+ });
+
+ this.heapMemoryLimitBytes = register(Gauge.build(prometheusPrefix + "topic_list_heap_memory_limit_bytes",
+ "Configured heap memory limit")
+ .create());
+ this.heapMemoryLimitBytes.set(maxHeapMemory);
+ this.otelHeapMemoryLimitGauge = openTelemetryMeter.gaugeBuilder("topic.list.heap.memory.limit")
+ .setUnit("By")
+ .setDescription("Configured heap memory limit")
+ .build();
+ this.otelHeapMemoryLimitGauge.set(maxHeapMemory);
+
+ this.directMemoryUsedBytes = register(Gauge.build(prometheusPrefix + "topic_list_direct_memory_used_bytes",
+ "Current direct memory used by topic listings")
+ .create()
+ .setChild(new Gauge.Child() {
+ @Override
+ public double get() {
+ return directMemoryLimiter.getAcquiredPermits();
+ }
+ }));
+ this.otelDirectMemoryUsedGauge = openTelemetryMeter.gaugeBuilder("topic.list.direct.memory.used")
+ .setUnit("By")
+ .setDescription("Current direct memory used by topic listings")
+ .buildWithCallback(observableDoubleMeasurement -> {
+ observableDoubleMeasurement.record(directMemoryLimiter.getAcquiredPermits());
+ });
+
+ this.directMemoryLimitBytes = register(Gauge.build(prometheusPrefix + "topic_list_direct_memory_limit_bytes",
+ "Configured direct memory limit")
+ .create());
+ this.directMemoryLimitBytes.set(maxDirectMemory);
+ this.otelDirectMemoryLimitGauge = openTelemetryMeter.gaugeBuilder("topic.list.direct.memory.limit")
+ .setUnit("By")
+ .setDescription("Configured direct memory limit")
+ .build();
+ this.otelDirectMemoryLimitGauge.set(maxDirectMemory);
+
+ this.heapQueueSize = register(Gauge.build(prometheusPrefix + "topic_list_heap_queue_size",
+ "Current heap memory limiter queue size")
+ .create()
+ .setChild(new Gauge.Child() {
+ @Override
+ public double get() {
+ return heapMemoryLimiter.getQueueSize();
+ }
+ }));
+ this.otelHeapQueueSize = openTelemetryMeter
+ .upDownCounterBuilder("topic.list.heap.queue.size")
+ .setDescription("Current heap memory limiter queue size")
+ .setUnit("1")
+ .buildWithCallback(observableLongMeasurement -> {
+ observableLongMeasurement.record(heapMemoryLimiter.getQueueSize());
+ });
+
+ this.heapQueueMaxSize = register(Gauge.build(prometheusPrefix + "topic_list_heap_queue_max_size",
+ "Maximum heap memory limiter queue size")
+ .create());
+ this.heapQueueMaxSize.set(maxHeapQueueSize);
+ LongGauge otelHeapQueueMaxSize = openTelemetryMeter
+ .gaugeBuilder("topic.list.heap.queue.max.size")
+ .setDescription("Maximum heap memory limiter queue size")
+ .setUnit("1")
+ .ofLongs()
+ .build();
+ otelHeapQueueMaxSize.set(maxHeapQueueSize);
+
+ this.directQueueSize = register(Gauge.build(prometheusPrefix + "topic_list_direct_queue_size",
+ "Current direct memory limiter queue size")
+ .create()
+ .setChild(new Gauge.Child() {
+ @Override
+ public double get() {
+ return directMemoryLimiter.getQueueSize();
+ }
+ }));
+ this.otelDirectQueueSize = openTelemetryMeter
+ .upDownCounterBuilder("topic.list.direct.queue.size")
+ .setDescription("Current direct memory limiter queue size")
+ .setUnit("1")
+ .buildWithCallback(observableLongMeasurement -> {
+ observableLongMeasurement.record(directMemoryLimiter.getQueueSize());
+ });
+
+ this.directQueueMaxSize = register(Gauge.build(prometheusPrefix + "topic_list_direct_queue_max_size",
+ "Maximum direct memory limiter queue size")
+ .create());
+ this.directQueueMaxSize.set(maxDirectQueueSize);
+ LongGauge otelDirectQueueMaxSize = openTelemetryMeter
+ .gaugeBuilder("topic.list.direct.queue.max.size")
+ .setDescription("Maximum direct memory limiter queue size")
+ .setUnit("1")
+ .ofLongs()
+ .build();
+ otelDirectQueueMaxSize.set(maxDirectQueueSize);
+
+ this.heapWaitTimeMs = register(Summary.build(prometheusPrefix + "topic_list_heap_wait_time_ms",
+ "Wait time for heap memory permits")
+ .quantile(0.50, 0.01)
+ .quantile(0.95, 0.01)
+ .quantile(0.99, 0.01)
+ .quantile(1, 0.01)
+ .create());
+ this.otelHeapWaitTime = openTelemetryMeter.histogramBuilder("topic.list.heap.wait.time")
+ .setUnit("s")
+ .setDescription("Wait time for heap memory permits")
+ .build();
+
+ this.directWaitTimeMs = register(Summary.build(prometheusPrefix + "topic_list_direct_wait_time_ms",
+ "Wait time for direct memory permits")
+ .quantile(0.50, 0.01)
+ .quantile(0.95, 0.01)
+ .quantile(0.99, 0.01)
+ .quantile(1, 0.01)
+ .create());
+ this.otelDirectWaitTime = openTelemetryMeter.histogramBuilder("topic.list.direct.wait.time")
+ .setUnit("s")
+ .setDescription("Wait time for direct memory permits")
+ .build();
+
+ this.heapTimeoutTotal = register(Counter.build(prometheusPrefix + "topic_list_heap_timeout_total",
+ "Total heap memory permit timeouts")
+ .create());
+ this.otelHeapTimeoutTotal = openTelemetryMeter.counterBuilder("topic.list.heap.timeout.total")
+ .setDescription("Total heap memory permit timeouts")
+ .setUnit("1")
+ .build();
+
+ this.directTimeoutTotal = register(Counter.build(prometheusPrefix + "topic_list_direct_timeout_total",
+ "Total direct memory permit timeouts")
+ .create());
+ this.otelDirectTimeoutTotal = openTelemetryMeter.counterBuilder("topic.list.direct.timeout.total")
+ .setDescription("Total direct memory permit timeouts")
+ .setUnit("1")
+ .build();
+ }
+
+ private <T extends Collector> T register(T collector) {
+ try {
+ collectorRegistry.register(collector);
+ } catch (Exception e) {
+ // ignore exception when registering a collector that is already registered
+ if (log.isDebugEnabled()) {
+ log.debug("Failed to register Prometheus collector {}", collector, e);
+ }
+ }
+ return collector;
+ }
+
+ private void unregister(Collector collector) {
+ try {
+ collectorRegistry.unregister(collector);
+ } catch (Exception e) {
+ // ignore exception when unregistering a collector that is not registered
+ if (log.isDebugEnabled()) {
+ log.debug("Failed to unregister Prometheus collector {}", collector, e);
+ }
+ }
+ }
+
+
+ @Override
+ protected void recordHeapWaitTime(long waitTimeNanos) {
+ if (waitTimeNanos == Long.MAX_VALUE) {
+ heapTimeoutTotal.inc();
+ otelHeapTimeoutTotal.add(1);
+ } else {
+ heapWaitTimeMs.observe(TimeUnit.NANOSECONDS.toMillis(waitTimeNanos));
+ otelHeapWaitTime.record(waitTimeNanos / 1_000_000_000.0d);
+ }
+ }
+
+ @Override
+ protected void recordDirectWaitTime(long waitTimeNanos) {
+ if (waitTimeNanos == Long.MAX_VALUE) {
+ directTimeoutTotal.inc();
+ otelDirectTimeoutTotal.add(1);
+ } else {
+ directWaitTimeMs.observe(TimeUnit.NANOSECONDS.toMillis(waitTimeNanos));
+ otelDirectWaitTime.record(waitTimeNanos / 1_000_000_000.0d);
+ }
+ }
+
+ @Override
+ public void close() {
+ super.close();
+ unregister(heapMemoryUsedBytes);
+ unregister(heapMemoryLimitBytes);
+ unregister(directMemoryUsedBytes);
+ unregister(directMemoryLimitBytes);
+ unregister(heapQueueSize);
+ unregister(heapQueueMaxSize);
+ unregister(directQueueSize);
+ unregister(directQueueMaxSize);
+ unregister(heapWaitTimeMs);
+ unregister(directWaitTimeMs);
+ unregister(heapTimeoutTotal);
+ unregister(directTimeoutTotal);
+ otelHeapMemoryUsedGauge.close();
+ otelDirectMemoryUsedGauge.close();
+ otelHeapQueueSize.close();
+ otelDirectQueueSize.close();
+ }
+
+ /**
+ * Estimate the heap memory size of a topic list.
+ * @param topicList the topic list to estimate
+ * @return the estimated heap memory size in bytes
+ */
+ public static long estimateTopicListSize(List<String> topicList) {
+ return topicList.stream()
+ .mapToLong(ByteBufUtil::utf8Bytes) // convert character count to bytes
+ // add 32 bytes overhead for each entry
+ // 16 bytes for object header, 16 bytes for java.lang.String fields
+ .map(n -> n + 32)
+ .sum();
+ }
Review Comment:
The comment on line 306 'convert character count to bytes' is misleading. `ByteBufUtil.utf8Bytes()` returns the UTF-8 encoded byte count, not a character count. The comment should be updated to clarify that it calculates UTF-8 encoded byte size.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "Denovo1998 (via GitHub)" <gi...@apache.org>.
Denovo1998 commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2432187651
##########
pulsar-common/src/main/java/org/apache/pulsar/common/semaphore/AsyncSemaphoreImpl.java:
##########
@@ -0,0 +1,286 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.common.semaphore;
+
+import io.netty.util.concurrent.DefaultThreadFactory;
+import java.util.Queue;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicLongFieldUpdater;
+import java.util.function.BooleanSupplier;
+import java.util.function.LongConsumer;
+import org.apache.pulsar.common.util.Runnables;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of AsyncSemaphore with timeout and queue size limits.
+ */
+public class AsyncSemaphoreImpl implements AsyncSemaphore, AutoCloseable {
+ private static final Logger log = LoggerFactory.getLogger(AsyncSemaphoreImpl.class);
+
+ private final AtomicLong availablePermits;
+ private final Queue<PendingRequest> queue;
+ private final long maxPermits;
+ private final long timeoutMillis;
+ private final ScheduledExecutorService executor;
+ private final boolean shutdownExecutor;
+ private final LongConsumer queueLatencyRecorder;
+ private final AtomicBoolean closed = new AtomicBoolean(false);
+ private final Runnable processQueueRunnable = Runnables.catchingAndLoggingThrowables(this::internalProcessQueue);
+
+ public AsyncSemaphoreImpl(long maxPermits, int maxQueueSize, long timeoutMillis) {
+ this(maxPermits, maxQueueSize, timeoutMillis, createExecutor(), true, null);
+ }
+
+ public AsyncSemaphoreImpl(long maxPermits, int maxQueueSize, long timeoutMillis,
+ ScheduledExecutorService executor, LongConsumer queueLatencyRecorder) {
+ this(maxPermits, maxQueueSize, timeoutMillis, executor, false, queueLatencyRecorder);
+ }
+
+ AsyncSemaphoreImpl(long maxPermits, int maxQueueSize, long timeoutMillis, ScheduledExecutorService executor,
+ boolean shutdownExecutor, LongConsumer queueLatencyRecorder) {
+ this.availablePermits = new AtomicLong(maxPermits);
+ this.maxPermits = maxPermits;
+ this.queue = new ArrayBlockingQueue<>(maxQueueSize);
+ this.timeoutMillis = timeoutMillis;
+ this.executor = executor;
+ this.shutdownExecutor = shutdownExecutor;
+ this.queueLatencyRecorder = queueLatencyRecorder;
+ }
+
+ private static ScheduledExecutorService createExecutor() {
+ return Executors.newSingleThreadScheduledExecutor(
+ new DefaultThreadFactory("async-semaphore-executor"));
+ }
+
+ @Override
+ public CompletableFuture<AsyncSemaphorePermit> acquire(long permits, BooleanSupplier isCancelled) {
+ return internalAcquire(permits, permits, isCancelled);
+ }
+
+ private CompletableFuture<AsyncSemaphorePermit> internalAcquire(long permits, long acquirePermits,
+ BooleanSupplier isCancelled) {
+ if (permits < 0) {
+ throw new IllegalArgumentException("Invalid permits value: " + permits);
+ }
+
+ CompletableFuture<AsyncSemaphorePermit> future = new CompletableFuture<>();
+
+ if (closed.get()) {
+ future.completeExceptionally(new PermitAcquireAlreadyClosedException("Semaphore is closed"));
+ return future;
+ }
+
+ PendingRequest request = new PendingRequest(permits, acquirePermits, future, isCancelled);
+ if (!queue.offer(request)) {
+ future.completeExceptionally(new PermitAcquireQueueFullException(
+ "Semaphore queue is full"));
+ return future;
+ }
+ // Schedule timeout
+ ScheduledFuture<?> timeoutTask = executor.schedule(() -> {
+ if (!request.future.isDone() && queue.remove(request)) {
+ // timeout is recorded with Long.MAX_VALUE as the age
+ recordQueueLatency(Long.MAX_VALUE);
+ // also record the time in the queue
+ recordQueueLatency(request.getAgeNanos());
+ future.completeExceptionally(new PermitAcquireTimeoutException(
+ "Permit acquisition timed out"));
+ // the next request might have smaller permits and that might be processed
+ processQueue();
+ }
+ }, timeoutMillis, TimeUnit.MILLISECONDS);
+ request.setTimeoutTask(timeoutTask);
+
+ processQueue();
+ return future;
+ }
+
+ private void recordQueueLatency(long ageNanos) {
+ if (queueLatencyRecorder != null) {
+ queueLatencyRecorder.accept(ageNanos);
+ }
+ }
+
+ @Override
+ public CompletableFuture<AsyncSemaphorePermit> update(AsyncSemaphorePermit permit, long newPermits,
+ BooleanSupplier isCancelled) {
+ if (newPermits < 0) {
+ throw new IllegalArgumentException("Invalid permits value: " + newPermits);
+ }
+ long oldPermits = permit.getPermits();
+ long additionalPermits = newPermits - oldPermits;
+ // mark the old permits as released without adding the permits to availablePermits
+ castToImplementation(permit).releasePermits();
+ if (additionalPermits > 0) {
+ return internalAcquire(newPermits, additionalPermits, isCancelled);
+ } else {
+ // new permits are less than the old ones, so we return the difference
+ availablePermits.addAndGet(-additionalPermits);
+ processQueue();
+ // return the new permits immediately
+ return CompletableFuture.completedFuture(new SemaphorePermit(newPermits));
+ }
+ }
+
+ @Override
+ public void release(AsyncSemaphorePermit permit) {
+ availablePermits.addAndGet(castToImplementation(permit).releasePermits());
+ processQueue();
+ }
+
+ @Override
+ public long getAvailablePermits() {
+ return availablePermits.get();
+ }
+
+ @Override
+ public long getAcquiredPermits() {
+ return maxPermits - availablePermits.get();
+ }
+
+ @Override
+ public int getQueueSize() {
+ return queue.size();
+ }
+
+ private SemaphorePermit castToImplementation(AsyncSemaphorePermit permit) {
+ if (permit instanceof SemaphorePermit semaphorePermit) {
+ return semaphorePermit;
+ } else {
+ throw new IllegalArgumentException("Invalid permit type");
+ }
+ }
+
+ private void processQueue() {
+ if (closed.get()) {
+ return;
+ }
+ executor.execute(processQueueRunnable);
+ }
+
+ private void internalProcessQueue() {
Review Comment:
Is there a race condition here?
- Line 187: current = 10
- Request needs 8 permits
- Another thread calls update(), reducing availablePermits to 3
- Line 205: Check passes (8 <= 10 using stale current)
- Line 206: availablePermits = 3 - 8 = -5 (negative!)
This violates the semaphore invariant, allowing more permits to be acquired than maxPermits.
##########
pip/pip-442.md:
##########
@@ -163,227 +189,409 @@ public interface AsyncDualMemoryLimiter {
HEAP_MEMORY, // For heap memory allocation
DIRECT_MEMORY // For direct memory allocation
}
-
+
/**
* Acquire permits for the specified memory size.
* Returned future completes when memory permits are available.
- * It will complete exceptionally with AsyncSemaphorePermitAcquireTimeoutException on timeout
- * and exceptionally with AsyncSemaphorePermitAcquireQueueFullException when queue full
+ * It will complete exceptionally with AsyncSemaphore.PermitAcquireTimeoutException on timeout
+ * and exceptionally with AsyncSemaphore.PermitAcquireQueueFullException when queue full
+ *
+ * @param memorySize the size of memory to acquire permits for
+ * @param limitType the type of memory limit (HEAP_MEMORY or DIRECT_MEMORY)
+ * @param isCancelled supplier that returns true if acquisition should be cancelled
* @return CompletableFuture that completes with permit when available
*/
- CompletableFuture<AsyncDualMemoryLimiterPermit> acquire(long memorySize, LimitType limitType);
+ CompletableFuture<AsyncDualMemoryLimiterPermit> acquire(long memorySize, LimitType limitType,
+ BooleanSupplier isCancelled);
/**
* Acquire or release permits for previously acquired permits by updating the requested memory size.
* Returns a future that completes when permits are available.
- * It will complete exceptionally with AsyncSemaphorePermitAcquireTimeoutException on timeout
- * and exceptionally with AsyncSemaphorePermitAcquireQueueFullException when queue full
+ * It will complete exceptionally with AsyncSemaphore.PermitAcquireTimeoutException on timeout
+ * and exceptionally with AsyncSemaphore.PermitAcquireQueueFullException when queue full
+ * The provided permit is released when the permits are successfully acquired and the returned updated
+ * permit replaces the old instance.
+ *
+ * @param permit the previously acquired permit to update
+ * @param newMemorySize the new memory size to update to
+ * @param isCancelled supplier that returns true if update should be cancelled
* @return CompletableFuture that completes with permit when available
*/
- CompletableFuture<AsyncDualMemoryLimiterPermit> update(AsyncDualMemoryLimiterPermit permit, long newMemorySize);
-
+ CompletableFuture<AsyncDualMemoryLimiterPermit> update(AsyncDualMemoryLimiterPermit permit, long newMemorySize,
+ BooleanSupplier isCancelled);
+
/**
* Release previously acquired permit.
* Must be called to prevent memory permit leaks.
+ *
+ * @param permit the permit to release
*/
void release(AsyncDualMemoryLimiterPermit permit);
+ /**
+ * Execute the specified function with acquired permits and release the permits after the returned future completes.
+ * @param memorySize memory size to acquire permits for
+ * @param limitType memory limit type to acquire permits for
+ * @param function function to execute with acquired permits
+ * @return result of the function
+ * @param <T> type of the CompletableFuture returned by the function
+ */
+ default <T> CompletableFuture<T> withAcquiredPermits(long memorySize, LimitType limitType,
+ BooleanSupplier isCancelled,
+ Function<AsyncDualMemoryLimiterPermit,
+ CompletableFuture<T>> function,
+ Function<Throwable, CompletableFuture<T>>
+ permitAcquireErrorHandler) {
+ return AsyncDualMemoryLimiterUtil.withPermitsFuture(acquire(memorySize, limitType, isCancelled), function,
+ permitAcquireErrorHandler, this::release);
+ }
+
+ /**
+ * Executed the specified function with updated permits and release the permits after the returned future completes.
+ * @param initialPermit initial permit to update
+ * @param newMemorySize new memory size to update to
+ * @param function function to execute with updated permits
+ * @return result of the function
+ * @param <T> type of the CompletableFuture returned by the function
+ */
+ default <T> CompletableFuture<T> withUpdatedPermits(AsyncDualMemoryLimiterPermit initialPermit, long newMemorySize,
+ BooleanSupplier isCancelled,
+ Function<AsyncDualMemoryLimiterPermit,
+ CompletableFuture<T>> function,
+ Function<Throwable, CompletableFuture<T>>
+ permitAcquireErrorHandler) {
+ return AsyncDualMemoryLimiterUtil.withPermitsFuture(update(initialPermit, newMemorySize, isCancelled), function,
+ permitAcquireErrorHandler, this::release);
+ }
+}
+```
+
+#### AsyncDualMemoryLimiterUtil Helper
+
+A utility class provides helper methods for common patterns:
+
+```java
+public class AsyncDualMemoryLimiterUtil {
+ /**
+ * Execute a function with acquired permits and ensure permits are released after completion.
+ * This method handles the lifecycle of permits - acquisition, usage, and release, including error cases.
+ *
+ * @param permitsFuture Future that will complete with the required permits
+ * @param function Function to execute once permits are acquired that returns a CompletableFuture
+ * @param permitAcquireErrorHandler Handler for permit acquisition errors that returns a CompletableFuture
+ * @param releaser Consumer that handles releasing the permits
+ * @param <T> The type of result returned by the function
+ * @return CompletableFuture that completes with the result of the function execution
+ */
+ public static <T> CompletableFuture<T> withPermitsFuture(
+ CompletableFuture<AsyncDualMemoryLimiterPermit>
+ permitsFuture,
+ Function<AsyncDualMemoryLimiterPermit,
+ CompletableFuture<T>> function,
+ Function<Throwable, CompletableFuture<T>>
+ permitAcquireErrorHandler,
+ Consumer<AsyncDualMemoryLimiterPermit> releaser) {
+ // implementation omitted from PIP document
+ }
+
+ /**
+ * Acquires permits and writes the command as a response to the channel.
+ * Releases the permits after the response has been written to the socket or if the write fails.
+ *
+ * @param ctx the channel handler context used for writing the response
+ * @param dualMemoryLimiter the memory limiter used to acquire and release memory permits
+ * @param isCancelled supplier that indicates if the permit acquisition should be cancelled
+ * @param command the base command to serialize and write to the channel
+ * @param permitAcquireErrorHandler handler for errors that occur during permit acquisition
+ * @return a future that completes when the command has been written to the channel's outbound buffer
+ */
+ public static CompletableFuture<Void> acquireDirectMemoryPermitsAndWriteAndFlush(ChannelHandlerContext ctx,
+ AsyncDualMemoryLimiter
+ dualMemoryLimiter,
+ BooleanSupplier isCancelled,
+ BaseCommand command,
+ Consumer<Throwable>
+ permitAcquireErrorHandler
+ ) {
+ // implementation omitted from PIP document
+ }
}
```
#### Integration Points
-**1. Heap Memory Limiting (Post-Retrieval)**
+**1. Heap Memory Limiting (Post-Retrieval) - Broker**
-In `ServerCnx.handleGetTopicsOfNamespace`:
+In `ServerCnx.handleGetTopicsOfNamespace`, the implementation uses the helper methods:
```java
-// Acquire a fixed amount of permits initially since it's not known how much memory will be used
-// This will ensure that the operation continues only after it has the initial permits
-// It would be possible to use statistics for initial estimate, but this is simpler and sufficient
-maxTopicListInFlightLimiter.acquire(INITIAL_SIZE, AsyncDualMemoryLimiter.LimitType.HEAP_MEMORY)
- .thenCompose(initialPermit -> {
- getBrokerService().pulsar().getNamespaceService().getListOfUserTopics(namespaceName, mode)
- .thenCompose(topics -> {
- // Estimate memory after retrieval and update the permits to reflect the actual size
- long estimatedSize = topics.stream().mapToInt(String::length).sum();
- return maxTopicListInFlightLimiter
- .update(initialPermit, estimatedSize)
- .thenApply(permit -> Pair.of(topics, permit));
- })
- .thenAccept(topicsAndPermit -> {
- try {
- // Process and send response
- ...
- } finally {
- maxTopicListInFlightLimiter.release(topicsAndPermit.getRight());
- }
+ private void internalHandleGetTopicsOfNamespace(String namespace, NamespaceName namespaceName, long requestId,
+ CommandGetTopicsOfNamespace.Mode mode,
+ Optional<String> topicsPattern, Optional<String> topicsHash,
+ Semaphore lookupSemaphore) {
+ BooleanSupplier isPermitRequestCancelled = () -> !ctx().channel().isActive();
+ maxTopicListInFlightLimiter.withAcquiredPermits(INITIAL_TOPIC_LIST_HEAP_PERMITS_SIZE,
+ AsyncDualMemoryLimiter.LimitType.HEAP_MEMORY, isPermitRequestCancelled, initialPermits -> {
+ return getBrokerService().pulsar().getNamespaceService().getListOfUserTopics(namespaceName, mode)
+ .thenAccept(topics -> {
+ long actualSize = topics.stream().mapToInt(String::length).sum();
+ maxTopicListInFlightLimiter.withUpdatedPermits(initialPermits, actualSize,
+ isPermitRequestCancelled, permits -> {
+ boolean filterTopics = false;
+ // filter system topic
+ List<String> filteredTopics = topics;
+
+ if (enableSubscriptionPatternEvaluation && topicsPattern.isPresent()) {
+ if (topicsPattern.get().length() <= maxSubscriptionPatternLength) {
+ filterTopics = true;
+ filteredTopics = TopicList.filterTopics(filteredTopics, topicsPattern.get(),
+ topicsPatternImplementation);
+ } else {
+ log.info("[{}] Subscription pattern provided [{}] was longer than "
+ + "maximum {}.", remoteAddress, topicsPattern.get(),
+ maxSubscriptionPatternLength);
+ }
+ }
+ String hash = TopicList.calculateHash(filteredTopics);
+ boolean hashUnchanged = topicsHash.isPresent() && topicsHash.get().equals(hash);
+ if (hashUnchanged) {
+ filteredTopics = Collections.emptyList();
+ }
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Received CommandGetTopicsOfNamespace for namespace "
+ + "[//{}] by {}, size:{}", remoteAddress, namespace,
+ requestId,
+ topics.size());
+ }
+ commandSender.sendGetTopicsOfNamespaceResponse(filteredTopics, hash, filterTopics,
+ !hashUnchanged, requestId, ex -> {
+ log.warn("[{}] Failed to acquire direct memory permits for "
+ + "GetTopicsOfNamespace: {}", remoteAddress, ex.getMessage());
+ commandSender.sendErrorResponse(requestId, ServerError.TooManyRequests,
+ "Cannot acquire permits for direct memory");
+ });
+ return CompletableFuture.completedFuture(null);
+ }, t -> {
+ log.warn("[{}] Failed to acquire heap memory permits for "
+ + "GetTopicsOfNamespace: {}", remoteAddress, t.getMessage());
+ writeAndFlush(Commands.newError(requestId, ServerError.TooManyRequests,
+ "Failed due to heap memory limit exceeded"));
+ return CompletableFuture.completedFuture(null);
+ });
+ }).whenComplete((__, ___) -> {
+ lookupSemaphore.release();
+ }).exceptionally(ex -> {
+ log.warn("[{}] Error GetTopicsOfNamespace for namespace [//{}] by {}", remoteAddress,
+ namespace, requestId);
+ commandSender.sendErrorResponse(requestId,
+ BrokerServiceException.getClientErrorCode(new ServerMetadataException(ex)),
+ ex.getMessage());
+ return null;
+ });
+ }, t -> {
+ log.warn("[{}] Failed to acquire initial heap memory permits for GetTopicsOfNamespace: {}",
+ remoteAddress, t.getMessage());
+ writeAndFlush(Commands.newError(requestId, ServerError.TooManyRequests,
+ "Failed due to heap memory limit exceeded"));
+ lookupSemaphore.release();
+ return CompletableFuture.completedFuture(null);
});
- ...
- // For exceptional paths, initialPermit would need to be released
+}
```
-**2. Direct Memory Limiting (Pre-Serialization)**
+**2. Direct Memory Limiting (Pre-Serialization) - Broker**
-Modified `CommandSender` implementation:
+Modified `PulsarCommandSenderImpl`:
```java
@Override
public void sendGetTopicsOfNamespaceResponse(List<String> topics, String topicsHash,
- boolean filtered, boolean changed, long requestId) {
+ boolean filtered, boolean changed, long requestId,
+ Consumer<Throwable> permitAcquireErrorHandler) {
BaseCommand command = Commands.newGetTopicsOfNamespaceResponseCommand(topics, topicsHash,
filtered, changed, requestId);
safeIntercept(command, cnx);
- acquireMaxTopicListInFlightPermitsAndWriteAndFlush(command);
+ acquireDirectMemoryPermitsAndWriteAndFlush(cnx.ctx(), maxTopicListInFlightLimiter, () -> !cnx.isActive(),
+ command, permitAcquireErrorHandler);
}
+```
+
+The utility method implementation:
-private void acquireMaxTopicListInFlightPermitsAndWriteAndFlush(BaseCommand command) {
+```java
+public static CompletableFuture<Void> acquireDirectMemoryPermitsAndWriteAndFlush(ChannelHandlerContext ctx,
+ AsyncDualMemoryLimiter
+ dualMemoryLimiter,
+ BooleanSupplier isCancelled,
+ BaseCommand command,
+ Consumer<Throwable>
+ permitAcquireErrorHandler
+) {
// Calculate serialized size before acquiring permits
int serializedSize = command.getSerializedSize();
// Acquire permits
- maxTopicListInFlightLimiter.acquire(serializedSize, AsyncDualMemoryLimiter.LimitType.DIRECT_MEMORY)
- .thenAcceptAsync(permits -> {
+ return dualMemoryLimiter.acquire(serializedSize, AsyncDualMemoryLimiter.LimitType.DIRECT_MEMORY, isCancelled)
+ .whenComplete((permits, t) -> {
+ if (t != null) {
+ permitAcquireErrorHandler.accept(t);
+ return;
+ }
try {
// Serialize the response
ByteBuf outBuf = Commands.serializeWithPrecalculatedSerializedSize(command, serializedSize);
// Write the response
- cnx.ctx().writeAndFlush(outBuf).addListener(future -> {
+ ctx.writeAndFlush(outBuf).addListener(future -> {
// Release permits after the response has been written to the socket
- maxTopicListInFlightLimiter.release(permits);
+ dualMemoryLimiter.release(permits);
});
- } catch (Exception e) {
+ } catch (Throwable e) {
// Return permits if an exception occurs before writeAndFlush is called successfully
- maxTopicListInFlightLimiter.release(permits);
+ dualMemoryLimiter.release(permits);
throw e;
}
- }, cnx.ctx().executor());
+ }).thenApply(__ -> null);
}
```
-**3. Watch Command Memory Control**
+**3. Watch Command Memory Control - Broker**
-Similar memory limiting patterns apply to watch commands:
+Similar memory limiting patterns apply to watch commands in `TopicListService`:
```java
-public void sendWatchTopicListSuccess(long requestId, long watcherId, String topicsHash, List<String> topics) {
- BaseCommand command = Commands.newWatchTopicListSuccess(requestId, watcherId, topicsHash, topics);
- acquireMaxTopicListInFlightPermitsAndWriteAndFlush(command);
-}
-
-public void sendWatchTopicListUpdate(long watcherId, List<String> newTopics, List<String> deletedTopics, String topicsHash) {
- BaseCommand command = Commands.newWatchTopicUpdate(watcherId, newTopics, deletedTopics, topicsHash);
- acquireMaxTopicListInFlightPermitsAndWriteAndFlush(command);
+public void sendTopicListUpdate(long watcherId, String topicsHash,
+ List<String> deletedTopics, List<String> newTopics) {
+ connection.getCommandSender().sendWatchTopicListUpdate(
+ watcherId, newTopics, deletedTopics, topicsHash,
+ t -> {
+ log.warn("[{}] Cannot acquire direct memory tokens for sending topic list update",
+ connection.toString(), t);
+ });
}
```
-**4. Proxy Reading Memory Control**
-
-On the Pulsar Proxy side, the problem is slightly different. The problem occurs when the proxy receives a `CommandGetTopicsOfNamespace` command, forwards it to a broker, and receives a response. The proxy needs to deserialize and serialize the response before sending it to the client.
-Memory is allocated for both deserialization and serialization.
-
-Solving this requires a slight modification to PulsarDecoder.
+**4. Proxy Memory Control**
-In `PulsarDecoder.channelRead`, it would be necessary to record the size of the incoming message:
+On the Pulsar Proxy side in `LookupProxyHandler`:
```java
- // Get a buffer that contains the full frame
- ByteBuf buffer = (ByteBuf) msg;
- try {
- // De-serialize the command
- int cmdSize = (int) buffer.readUnsignedInt();
- cmd.parseFrom(buffer, cmdSize);
-```
+private void internalPerformGetTopicsOfNamespace(long clientRequestId, String namespaceName, ClientCnx clientCnx,
+ ByteBuf command, long requestId) {
+ BooleanSupplier isPermitRequestCancelled = () -> !proxyConnection.ctx().channel().isActive();
+ maxTopicListInFlightLimiter.withAcquiredPermits(INITIAL_TOPIC_LIST_HEAP_PERMITS_SIZE,
+ AsyncDualMemoryLimiter.LimitType.HEAP_MEMORY, isPermitRequestCancelled, initialPermits -> {
+ return clientCnx.newGetTopicsOfNamespace(command, requestId).whenComplete((r, t) -> {
+ if (t != null) {
+ log.warn("[{}] Failed to get TopicsOfNamespace {}: {}", clientAddress, namespaceName,
+ t.getMessage());
+ writeAndFlush(Commands.newError(clientRequestId, getServerError(t), t.getMessage()));
+ } else {
+ long actualSize =
+ r.getNonPartitionedOrPartitionTopics().stream().mapToInt(String::length).sum();
+ maxTopicListInFlightLimiter.withUpdatedPermits(initialPermits, actualSize,
+ isPermitRequestCancelled, permits -> {
+ return handleWritingGetTopicsResponse(clientRequestId, r, isPermitRequestCancelled);
+ }, t2 -> {
+ log.warn("[{}] Failed to acquire actual heap memory permits for "
+ + "GetTopicsOfNamespace: {}", clientAddress, t2.getMessage());
+ writeAndFlush(Commands.newError(clientRequestId, ServerError.TooManyRequests,
+ "Failed due to heap memory limit exceeded"));
+
+ return CompletableFuture.completedFuture(null);
+ });
+ }
+ }).thenApply(__ -> null);
+ }, t -> {
+ log.warn("[{}] Failed to acquire initial heap memory permits for GetTopicsOfNamespace: {}",
+ clientAddress, t.getMessage());
+ writeAndFlush(Commands.newError(clientRequestId, ServerError.TooManyRequests,
+ "Failed due to heap memory limit exceeded"));
+
+ return CompletableFuture.completedFuture(null);
+ }).exceptionally(ex -> {
+ writeAndFlush(Commands.newError(clientRequestId, getServerError(ex), ex.getMessage()));
+ return null;
+ });
+}
-It could be modified to store the `cmdSize` in a field instead of a local variable:
-```java
- protected int cmdSize;
-...
- // Get a buffer that contains the full frame
- ByteBuf buffer = (ByteBuf) msg;
- try {
- // De-serialize the command
- cmdSize = (int) buffer.readUnsignedInt();
- cmd.parseFrom(buffer, cmdSize);
+private CompletableFuture<Void> handleWritingGetTopicsResponse(long clientRequestId, GetTopicsResult r,
+ BooleanSupplier isCancelled) {
+ BaseCommand responseCommand = Commands.newGetTopicsOfNamespaceResponseCommand(
+ r.getNonPartitionedOrPartitionTopics(), r.getTopicsHash(), r.isFiltered(),
+ r.isChanged(), clientRequestId);
+ return acquireDirectMemoryPermitsAndWriteAndFlush(proxyConnection.ctx(), maxTopicListInFlightLimiter,
+ isCancelled, responseCommand, t -> {
+ log.warn("[{}] Failed to acquire actual direct memory permits for GetTopicsOfNamespace: {}",
+ clientAddress, t.getMessage());
+ writeAndFlush(Commands.newError(clientRequestId, ServerError.TooManyRequests,
+ "Failed due to heap memory limit exceeded"));
Review Comment:
```java
"Failed due to direct memory limit exceeded"
```
##########
pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/LookupProxyHandler.java:
##########
@@ -357,6 +358,62 @@ private void performGetTopicsOfNamespace(long clientRequestId,
});
}
+ private void internalPerformGetTopicsOfNamespace(long clientRequestId, String namespaceName, ClientCnx clientCnx,
+ ByteBuf command, long requestId) {
+ BooleanSupplier isPermitRequestCancelled = () -> !proxyConnection.ctx().channel().isActive();
+ maxTopicListInFlightLimiter.withAcquiredPermits(INITIAL_TOPIC_LIST_HEAP_PERMITS_SIZE,
+ AsyncDualMemoryLimiter.LimitType.HEAP_MEMORY, isPermitRequestCancelled, initialPermits -> {
+ return clientCnx.newGetTopicsOfNamespace(command, requestId).whenComplete((r, t) -> {
+ if (t != null) {
+ log.warn("[{}] Failed to get TopicsOfNamespace {}: {}", clientAddress, namespaceName,
+ t.getMessage());
+ writeAndFlush(Commands.newError(clientRequestId, getServerError(t), t.getMessage()));
+ } else {
+ long actualSize =
+ r.getNonPartitionedOrPartitionTopics().stream()
+ .mapToInt(ByteBufUtil::utf8Bytes) // convert character count to bytes
+ .map(n -> n + 32) // add 32 bytes overhead for each entry
Review Comment:
Can be changed to
```java
.mapToLong(topic -> ByteBufUtil.utf8Bytes(topic) + 32) // UTF-8 bytes + 32 bytes overhead per entry
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2429766441
##########
pulsar-broker-common/src/main/java/org/apache/pulsar/broker/topiclistlimit/TopicListMemoryLimiter.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.topiclistlimit;
+
+import io.opentelemetry.api.metrics.DoubleGauge;
+import io.opentelemetry.api.metrics.DoubleHistogram;
+import io.opentelemetry.api.metrics.LongCounter;
+import io.opentelemetry.api.metrics.LongGauge;
+import io.opentelemetry.api.metrics.Meter;
+import io.opentelemetry.api.metrics.ObservableDoubleGauge;
+import io.opentelemetry.api.metrics.ObservableLongUpDownCounter;
+import io.prometheus.client.Collector;
+import io.prometheus.client.CollectorRegistry;
+import io.prometheus.client.Counter;
+import io.prometheus.client.Gauge;
+import io.prometheus.client.Summary;
+import java.util.concurrent.TimeUnit;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.common.semaphore.AsyncDualMemoryLimiterImpl;
+import org.apache.pulsar.common.semaphore.AsyncSemaphore;
+
+/**
+ * Topic list memory limiter that exposes both Prometheus metrics and OpenTelemetry metrics.
+ */
+@Slf4j
+public class TopicListMemoryLimiter extends AsyncDualMemoryLimiterImpl {
+ private final CollectorRegistry collectorRegistry;
+ private final Gauge heapMemoryUsedBytes;
+ private final Gauge heapMemoryLimitBytes;
+ private final Gauge directMemoryUsedBytes;
+ private final Gauge directMemoryLimitBytes;
+ private final Gauge heapQueueSize;
+ private final Gauge heapQueueMaxSize;
+ private final Gauge directQueueSize;
+ private final Gauge directQueueMaxSize;
+ private final Summary heapWaitTimeMs;
+ private final Summary directWaitTimeMs;
+ private final Counter heapTimeoutTotal;
+ private final Counter directTimeoutTotal;
+ private final ObservableDoubleGauge otelHeapMemoryUsedGauge;
+ private final DoubleGauge otelHeapMemoryLimitGauge;
+ private final ObservableDoubleGauge otelDirectMemoryUsedGauge;
+ private final DoubleGauge otelDirectMemoryLimitGauge;
+ private final ObservableLongUpDownCounter otelHeapQueueSize;
+ private final ObservableLongUpDownCounter otelDirectQueueSize;
+ private final DoubleHistogram otelHeapWaitTime;
+ private final DoubleHistogram otelDirectWaitTime;
+ private final LongCounter otelHeapTimeoutTotal;
+ private final LongCounter otelDirectTimeoutTotal;
+
+ public TopicListMemoryLimiter(CollectorRegistry collectorRegistry, String prometheusPrefix,
+ Meter openTelemetryMeter,
+ long maxHeapMemory, int maxHeapQueueSize,
+ long heapTimeoutMillis, long maxDirectMemory, int maxDirectQueueSize,
+ long directTimeoutMillis) {
+ super(maxHeapMemory, maxHeapQueueSize, heapTimeoutMillis, maxDirectMemory, maxDirectQueueSize,
+ directTimeoutMillis);
+ this.collectorRegistry = collectorRegistry;
+
+ AsyncSemaphore heapMemoryLimiter = getLimiter(LimitType.HEAP_MEMORY);
+ AsyncSemaphore directMemoryLimiter = getLimiter(LimitType.DIRECT_MEMORY);
+
+ this.heapMemoryUsedBytes = register(Gauge.build(prometheusPrefix + "topic_list_heap_memory_used_bytes",
+ "Current heap memory used by topic listings")
+ .create()
+ .setChild(new Gauge.Child() {
+ @Override
+ public double get() {
+ return heapMemoryLimiter.getAcquiredPermits();
+ }
+ }));
+ this.otelHeapMemoryUsedGauge = openTelemetryMeter.gaugeBuilder("topic.list.heap.memory.used")
+ .setUnit("By")
+ .setDescription("Current heap memory used by topic listings")
+ .buildWithCallback(observableDoubleMeasurement -> {
+ observableDoubleMeasurement.record(heapMemoryLimiter.getAcquiredPermits());
+ });
+
+ this.heapMemoryLimitBytes = register(Gauge.build(prometheusPrefix + "topic_list_heap_memory_limit_bytes",
+ "Configured heap memory limit")
+ .create());
+ this.heapMemoryLimitBytes.set(maxHeapMemory);
+ this.otelHeapMemoryLimitGauge = openTelemetryMeter.gaugeBuilder("topic.list.heap.memory.limit")
+ .setUnit("By")
+ .setDescription("Configured heap memory limit")
+ .build();
+ this.otelHeapMemoryLimitGauge.set(maxHeapMemory);
+
+ this.directMemoryUsedBytes = register(Gauge.build(prometheusPrefix + "topic_list_direct_memory_used_bytes",
+ "Current direct memory used by topic listings")
+ .create()
+ .setChild(new Gauge.Child() {
+ @Override
+ public double get() {
+ return directMemoryLimiter.getAcquiredPermits();
+ }
+ }));
+ this.otelDirectMemoryUsedGauge = openTelemetryMeter.gaugeBuilder("topic.list.direct.memory.used")
+ .setUnit("By")
+ .setDescription("Current direct memory used by topic listings")
+ .buildWithCallback(observableDoubleMeasurement -> {
+ observableDoubleMeasurement.record(directMemoryLimiter.getAcquiredPermits());
+ });
+
+ this.directMemoryLimitBytes = register(Gauge.build(prometheusPrefix + "topic_list_direct_memory_limit_bytes",
+ "Configured direct memory limit")
+ .create());
+ this.directMemoryLimitBytes.set(maxDirectMemory);
+ this.otelDirectMemoryLimitGauge = openTelemetryMeter.gaugeBuilder("topic.list.direct.memory.limit")
+ .setUnit("By")
+ .setDescription("Configured direct memory limit")
+ .build();
+ this.otelDirectMemoryLimitGauge.set(maxDirectMemory);
+
+ this.heapQueueSize = register(Gauge.build(prometheusPrefix + "topic_list_heap_queue_size",
+ "Current heap memory limiter queue size")
+ .create()
+ .setChild(new Gauge.Child() {
+ @Override
+ public double get() {
+ return heapMemoryLimiter.getQueueSize();
+ }
+ }));
+ this.otelHeapQueueSize = openTelemetryMeter
+ .upDownCounterBuilder("topic.list.heap.queue.size")
+ .setDescription("Current heap memory limiter queue size")
+ .setUnit("1")
+ .buildWithCallback(observableLongMeasurement -> {
+ observableLongMeasurement.record(heapMemoryLimiter.getQueueSize());
+ });
+
+ this.heapQueueMaxSize = register(Gauge.build(prometheusPrefix + "topic_list_heap_queue_max_size",
+ "Maximum heap memory limiter queue size")
+ .create());
+ this.heapQueueMaxSize.set(maxHeapQueueSize);
+ LongGauge otelHeapQueueMaxSize = openTelemetryMeter
+ .gaugeBuilder("topic.list.heap.queue.max.size")
+ .setDescription("Maximum heap memory limiter queue size")
+ .setUnit("1")
+ .ofLongs()
+ .build();
+ otelHeapQueueMaxSize.set(maxHeapQueueSize);
+
+ this.directQueueSize = register(Gauge.build(prometheusPrefix + "topic_list_direct_queue_size",
+ "Current direct memory limiter queue size")
+ .create()
+ .setChild(new Gauge.Child() {
+ @Override
+ public double get() {
+ return directMemoryLimiter.getQueueSize();
+ }
+ }));
+ this.otelDirectQueueSize = openTelemetryMeter
+ .upDownCounterBuilder("topic.list.direct.queue.size")
+ .setDescription("Current direct memory limiter queue size")
+ .setUnit("1")
+ .buildWithCallback(observableLongMeasurement -> {
+ observableLongMeasurement.record(directMemoryLimiter.getQueueSize());
+ });
+
+ this.directQueueMaxSize = register(Gauge.build(prometheusPrefix + "topic_list_direct_queue_max_size",
+ "Maximum direct memory limiter queue size")
+ .create());
+ this.directQueueMaxSize.set(maxDirectQueueSize);
+ LongGauge otelDirectQueueMaxSize = openTelemetryMeter
+ .gaugeBuilder("topic.list.direct.queue.max.size")
+ .setDescription("Maximum direct memory limiter queue size")
+ .setUnit("1")
+ .ofLongs()
+ .build();
+ otelDirectQueueMaxSize.set(maxDirectQueueSize);
+
+ this.heapWaitTimeMs = register(Summary.build(prometheusPrefix + "topic_list_heap_wait_time_ms",
+ "Wait time for heap memory permits")
+ .quantile(0.50, 0.01)
+ .quantile(0.95, 0.01)
+ .quantile(0.99, 0.01)
+ .quantile(1, 0.01)
+ .create());
+ this.otelHeapWaitTime = openTelemetryMeter.histogramBuilder("topic.list.heap.wait.time.ms")
+ .setUnit("s")
+ .setDescription("Wait time for heap memory permits")
+ .build();
+
+ this.directWaitTimeMs = register(Summary.build(prometheusPrefix + "topic_list_direct_wait_time_ms",
+ "Wait time for direct memory permits")
+ .quantile(0.50, 0.01)
+ .quantile(0.95, 0.01)
+ .quantile(0.99, 0.01)
+ .quantile(1, 0.01)
+ .create());
+ this.otelDirectWaitTime = openTelemetryMeter.histogramBuilder("topic.list.direct.wait.time.ms")
+ .setUnit("s")
Review Comment:
Thanks for pointing this out. I'll remove the ".ms" suffix which is wrong as you have pointed out. In Otel, the metrics are in seconds. That's part of the specification.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2427137054
##########
org/apache/pulsar/common/util/memory/AsyncSemaphore.java:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.common.util.memory;
+
+import java.util.concurrent.CompletableFuture;
+
+/**
+ * Interface for asynchronous semaphore operations.
+ * Used for managing concurrent access to limited resources.
+ */
+public interface AsyncSemaphore {
Review Comment:
Oh yes, now I realized the problem. Deleted the files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2466068257
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/NamespacesBase.java:
##########
@@ -165,8 +173,66 @@ protected CompletableFuture<Void> internalCreateNamespace(Policies policies) {
.thenAccept(__ -> log.info("[{}] Created namespace {}", clientAppId(), namespaceName));
}
- protected CompletableFuture<List<String>> internalGetListOfTopics(Policies policies,
+ protected CompletableFuture<List<String>> internalGetListOfTopics(AsyncResponse response, Policies policies,
CommandGetTopicsOfNamespace.Mode mode) {
+ // Use maxTopicListInFlightLimiter to limit inflight get topic listing responses
+ // to avoid OOME caused by a lot of clients using HTTP service lookups to list topics
+ AsyncDualMemoryLimiterImpl maxTopicListInFlightLimiter =
+ pulsar().getBrokerService().getMaxTopicListInFlightLimiter();
+ TopicListSizeResultCache.ResultHolder listSizeHolder = pulsar().getBrokerService().getTopicListSizeResultCache()
+ .getTopicListSize(namespaceName.toString(), mode);
+ // setup the permit cancellation function
+ AtomicBoolean permitRequestCancelled = new AtomicBoolean(false);
+ BooleanSupplier isPermitRequestCancelled = permitRequestCancelled::get;
+ // add callback that releases permits when the response completes
+ AtomicReference<AsyncDualMemoryLimiter.AsyncDualMemoryLimiterPermit> initialPermitsRef =
+ new AtomicReference<>();
+ AtomicReference<AsyncDualMemoryLimiter.AsyncDualMemoryLimiterPermit> permitsRef = new AtomicReference<>();
+ response.register(new CompletionCallback() {
+ @Override
+ public void onComplete(Throwable throwable) {
+ if (throwable != null) {
+ // for failed request
+ // handle resetting the TopicListSizeResultCache.ResultHolder
+ listSizeHolder.resetIfInitializing();
+ // cancel any pending permit request
+ permitRequestCancelled.set(true);
+ }
+ AsyncDualMemoryLimiter.AsyncDualMemoryLimiterPermit initialPermit = initialPermitsRef.get();
+ if (initialPermit != null) {
+ maxTopicListInFlightLimiter.release(initialPermit);
+ }
+ AsyncDualMemoryLimiter.AsyncDualMemoryLimiterPermit permits = permitsRef.get();
+ if (permits != null) {
+ maxTopicListInFlightLimiter.release(permits);
+ }
+ }
+ });
+ return listSizeHolder.getSizeAsync().thenCompose(initialSize -> maxTopicListInFlightLimiter.acquire(initialSize,
+ AsyncDualMemoryLimiter.LimitType.HEAP_MEMORY, isPermitRequestCancelled).exceptionally(t -> {
+ throw new CompletionException(
+ new RestException(Status.TOO_MANY_REQUESTS, "Failed due to heap memory limit exceeded"));
+ }).thenCompose(initialPermits -> {
+ initialPermitsRef.set(initialPermits);
+ // perform the actual get list of topics operation
+ return doInternalGetListOfTopics(policies, mode).thenCompose(topicList -> {
+ long actualSize = TopicListMemoryLimiter.estimateTopicListSize(topicList);
+ listSizeHolder.updateSize(actualSize);
+ return maxTopicListInFlightLimiter.update(initialPermits, actualSize, isPermitRequestCancelled)
Review Comment:
similar response.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "poorbarcode (via GitHub)" <gi...@apache.org>.
poorbarcode commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2501961567
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/TopicListService.java:
##########
@@ -101,6 +112,7 @@ public void accept(String topicName, NotificationType notificationType) {
private final boolean enableSubscriptionPatternEvaluation;
private final int maxSubscriptionPatternLength;
private final ConcurrentLongHashMap<CompletableFuture<TopicListWatcher>> watchers;
+ private final Backoff retryBackoff;
public TopicListService(PulsarService pulsar, ServerCnx connection,
Review Comment:
> That doesn't hold since the solution will retry with back off in case of errors.
It absolutely holds, because you set a `MAX_RETRY_COUNT (default value is 10) `, right?
> In case of running out of permits, there will be infinite retries (until the watcher is closed by closing it's connection).
This new change can solve the issue I mentioned. But I still think that we'd better remove this limitation because the topic change event uses less memory, and the event will rarely occur
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "poorbarcode (via GitHub)" <gi...@apache.org>.
poorbarcode commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2501961567
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/TopicListService.java:
##########
@@ -101,6 +112,7 @@ public void accept(String topicName, NotificationType notificationType) {
private final boolean enableSubscriptionPatternEvaluation;
private final int maxSubscriptionPatternLength;
private final ConcurrentLongHashMap<CompletableFuture<TopicListWatcher>> watchers;
+ private final Backoff retryBackoff;
public TopicListService(PulsarService pulsar, ServerCnx connection,
Review Comment:
> That doesn't hold since the solution will retry with back off in case of errors.
It absolutely holds, because you set a `MAX_RETRY_COUNT (default value is 10) `, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2504054507
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/TopicListService.java:
##########
@@ -101,6 +112,7 @@ public void accept(String topicName, NotificationType notificationType) {
private final boolean enableSubscriptionPatternEvaluation;
private final int maxSubscriptionPatternLength;
private final ConcurrentLongHashMap<CompletableFuture<TopicListWatcher>> watchers;
+ private final Backoff retryBackoff;
public TopicListService(PulsarService pulsar, ServerCnx connection,
Review Comment:
Topic list watcher changes have been reverted. @poorbarcode PTAL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2466335166
##########
pulsar-common/src/main/java/org/apache/pulsar/common/semaphore/AsyncSemaphoreImpl.java:
##########
@@ -0,0 +1,328 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.common.semaphore;
+
+import io.netty.util.concurrent.DefaultThreadFactory;
+import java.util.Queue;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicLongFieldUpdater;
+import java.util.function.BooleanSupplier;
+import java.util.function.LongConsumer;
+import org.apache.pulsar.common.util.Runnables;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of AsyncSemaphore with timeout and queue size limits.
+ */
+public class AsyncSemaphoreImpl implements AsyncSemaphore, AutoCloseable {
+ private static final Logger log = LoggerFactory.getLogger(AsyncSemaphoreImpl.class);
+
+ private final AtomicLong availablePermits;
+ private final Queue<PendingRequest> queue;
+ private final long maxPermits;
+ private final long timeoutMillis;
+ private final ScheduledExecutorService executor;
+ private final boolean shutdownExecutor;
+ private final LongConsumer queueLatencyRecorder;
+ private final AtomicBoolean closed = new AtomicBoolean(false);
+ private final Runnable processQueueRunnable = Runnables.catchingAndLoggingThrowables(this::internalProcessQueue);
+
+ /**
+ * Creates an AsyncSemaphoreImpl with the given parameters.
+ * @param maxPermits max number of permits available for acquisition, set to <= 0 for unbounded semaphore (not
+ * recommended)
+ * @param maxQueueSize max number of requests that can be queued, set to <= 0 for unbounded queue (not recommended)
+ * @param timeoutMillis timeout in milliseconds for acquiring permits
+ */
+ public AsyncSemaphoreImpl(long maxPermits, int maxQueueSize, long timeoutMillis) {
+ this(maxPermits, maxQueueSize, timeoutMillis, maxPermits > 0 ? createExecutor() : null, maxPermits > 0, null);
+ }
+
+ /**
+ * Creates an AsyncSemaphoreImpl with the given parameters.
+ * @param maxPermits max number of permits available for acquisition, set to <= 0 for unbounded semaphore (not
+ * recommended)
+ * @param maxQueueSize max number of requests that can be queued, set to <= 0 for unbounded queue (not recommended)
+ * @param timeoutMillis timeout in milliseconds for acquiring permits
+ * @param executor executor service to use for scheduling timeouts, it is expected to be single threaded
+ * @param queueLatencyRecorder consumer to record queue latency, Long.MAX_VALUE is used for requests that timed out
+ */
+ public AsyncSemaphoreImpl(long maxPermits, int maxQueueSize, long timeoutMillis,
+ ScheduledExecutorService executor, LongConsumer queueLatencyRecorder) {
+ this(maxPermits, maxQueueSize, timeoutMillis, executor, false, queueLatencyRecorder);
+ }
+
+ AsyncSemaphoreImpl(long maxPermits, int maxQueueSize, long timeoutMillis, ScheduledExecutorService executor,
+ boolean shutdownExecutor, LongConsumer queueLatencyRecorder) {
+ this.availablePermits = new AtomicLong(maxPermits);
+ this.maxPermits = maxPermits;
+ this.queue = maxQueueSize > 0 ? new ArrayBlockingQueue<>(maxQueueSize) : new LinkedBlockingQueue<>();
+ this.timeoutMillis = timeoutMillis;
+ this.executor = executor;
+ this.shutdownExecutor = shutdownExecutor;
+ this.queueLatencyRecorder = queueLatencyRecorder;
+ }
+
+ private static ScheduledExecutorService createExecutor() {
+ return Executors.newSingleThreadScheduledExecutor(
+ new DefaultThreadFactory("async-semaphore-executor"));
+ }
+
+ @Override
+ public CompletableFuture<AsyncSemaphorePermit> acquire(long permits, BooleanSupplier isCancelled) {
+ return internalAcquire(permits, permits, isCancelled);
+ }
+
+ private CompletableFuture<AsyncSemaphorePermit> internalAcquire(long permits, long acquirePermits,
+ BooleanSupplier isCancelled) {
+ if (!isPermitsValidForAcquiring(permits)) {
Review Comment:
ok, will add separate exception messages for these cases
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2466350154
##########
pulsar-broker-common/src/main/java/org/apache/pulsar/broker/topiclistlimit/TopicListSizeResultCache.java:
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.topiclistlimit;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReference;
+import org.apache.pulsar.common.api.proto.CommandGetTopicsOfNamespace;
+
+/**
+ * A cache for storing and managing topic list size estimates in namespaces.
+ * This class provides functionality to:
+ * - Cache and retrieve topic list size estimates for different namespaces and modes
+ * - Handle concurrent requests for topic list sizes efficiently
+ * - Maintain and update size estimates based on actual topic list sizes
+ * - Prevent thundering herd problems when multiple concurrent requests for a namespace are made without
+ * a previous size estimate
+ * The cache uses namespace name and topic list mode (PERSISTENT/NON_PERSISTENT/ALL) as keys
+ * and maintains size estimates that are refined with actual usage.
+ */
+public class TopicListSizeResultCache {
+ // 10kB initial estimate for topic list heap size
+ private static final long INITIAL_TOPIC_LIST_HEAP_SIZE = 10 * 1024;
+
+ private Cache<CacheKey, ResultHolder> topicListSizeCache = Caffeine.newBuilder()
+ .maximumSize(10000)
+ .build();
+
+ record CacheKey(String namespaceName, CommandGetTopicsOfNamespace.Mode mode) {
+ }
+
+ /**
+ * Holds the topic list size estimate and future for the topic list size.
+ * The size is returned by calling {@link #getSizeAsync()} method which is asynchronous.
+ * The estimate is updated by calling {@link #updateSize(long)} method.
+ */
+ public static class ResultHolder {
+ private final AtomicReference<CompletableFuture<Long>> topicListSizeFuture =
+ new AtomicReference<>(null);
+ private final AtomicLong existingSizeRef = new AtomicLong(-1L);
+
+ /**
+ * Get the topic list size estimate. The first request will return the initial estimate
+ * and update the estimate based on the returned size of the topic list. Other concurrent requests
+ * will wait for the first request to complete and use the estimate of the first request.
+ * Subsequent requests will use estimate which gets updated based on the returned size of the topic list
+ * of each request.
+ * @return a future that will return the topic list size estimate
+ */
+ public CompletableFuture<Long> getSizeAsync() {
+ if (topicListSizeFuture.compareAndSet(null, new CompletableFuture<>())) {
+ // let the first request proceed with the initial estimate
+ return CompletableFuture.completedFuture(INITIAL_TOPIC_LIST_HEAP_SIZE);
+ } else {
+ // all other requests wait for the first one to complete
+ return topicListSizeFuture.get();
+ }
+ }
+
+ /**
+ * Update the topic list size estimate. The new estimated size will be updated by calculating the average
+ * of the existing and the new size. If the difference between the new and the existing size is less than 1,
+ * no update will be done.
+ * @param actualSize the actual size of the topic list
+ */
+ public void updateSize(long actualSize) {
Review Comment:
Adding comments
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2466056060
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/ServerCnx.java:
##########
@@ -2610,6 +2581,93 @@ protected void handleGetTopicsOfNamespace(CommandGetTopicsOfNamespace commandGet
}
}
+ private void internalHandleGetTopicsOfNamespace(String namespace, NamespaceName namespaceName, long requestId,
+ CommandGetTopicsOfNamespace.Mode mode,
+ Optional<String> topicsPattern, Optional<String> topicsHash,
+ Semaphore lookupSemaphore) {
+ BooleanSupplier isPermitRequestCancelled = () -> !ctx().channel().isActive();
+ TopicListSizeResultCache.ResultHolder
+ listSizeHolder = service.getTopicListSizeResultCache().getTopicListSize(namespaceName.toString(), mode);
+ listSizeHolder.getSizeAsync().thenAccept(initialSize -> {
+ maxTopicListInFlightLimiter.withAcquiredPermits(initialSize,
+ AsyncDualMemoryLimiter.LimitType.HEAP_MEMORY, isPermitRequestCancelled, initialPermits -> {
+ return getBrokerService().pulsar().getNamespaceService()
+ .getListOfUserTopics(namespaceName, mode)
+ .thenAccept(topics -> {
+ long actualSize = TopicListMemoryLimiter.estimateTopicListSize(topics);
+ listSizeHolder.updateSize(actualSize);
+ maxTopicListInFlightLimiter.withUpdatedPermits(initialPermits, actualSize,
Review Comment:
> Instead of updating and acquiring additional permits if the actual size is larger than estimated, we'd better improve as follows:
>
> * The current request does not need to acquire permits again; only updating the permits that were borrowed out is enough, even though the permits were over-acquired
> * Updates the permits that were borrowed out, to limit the following permits acquisition;
This is already encapsulated in the logic of the `update` implementation. Please check the implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2436951451
##########
pulsar-broker-common/src/main/java/org/apache/pulsar/broker/topiclistlimit/TopicListMemoryLimiter.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.topiclistlimit;
+
+import io.opentelemetry.api.metrics.DoubleGauge;
+import io.opentelemetry.api.metrics.DoubleHistogram;
+import io.opentelemetry.api.metrics.LongCounter;
+import io.opentelemetry.api.metrics.LongGauge;
+import io.opentelemetry.api.metrics.Meter;
+import io.opentelemetry.api.metrics.ObservableDoubleGauge;
+import io.opentelemetry.api.metrics.ObservableLongUpDownCounter;
+import io.prometheus.client.Collector;
+import io.prometheus.client.CollectorRegistry;
+import io.prometheus.client.Counter;
+import io.prometheus.client.Gauge;
+import io.prometheus.client.Summary;
+import java.util.concurrent.TimeUnit;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.common.semaphore.AsyncDualMemoryLimiterImpl;
+import org.apache.pulsar.common.semaphore.AsyncSemaphore;
+
+/**
+ * Topic list memory limiter that exposes both Prometheus metrics and OpenTelemetry metrics.
+ */
+@Slf4j
+public class TopicListMemoryLimiter extends AsyncDualMemoryLimiterImpl {
+ private final CollectorRegistry collectorRegistry;
+ private final Gauge heapMemoryUsedBytes;
+ private final Gauge heapMemoryLimitBytes;
+ private final Gauge directMemoryUsedBytes;
+ private final Gauge directMemoryLimitBytes;
+ private final Gauge heapQueueSize;
+ private final Gauge heapQueueMaxSize;
+ private final Gauge directQueueSize;
+ private final Gauge directQueueMaxSize;
+ private final Summary heapWaitTimeMs;
+ private final Summary directWaitTimeMs;
+ private final Counter heapTimeoutTotal;
+ private final Counter directTimeoutTotal;
+ private final ObservableDoubleGauge otelHeapMemoryUsedGauge;
+ private final DoubleGauge otelHeapMemoryLimitGauge;
+ private final ObservableDoubleGauge otelDirectMemoryUsedGauge;
+ private final DoubleGauge otelDirectMemoryLimitGauge;
+ private final ObservableLongUpDownCounter otelHeapQueueSize;
+ private final ObservableLongUpDownCounter otelDirectQueueSize;
+ private final DoubleHistogram otelHeapWaitTime;
+ private final DoubleHistogram otelDirectWaitTime;
+ private final LongCounter otelHeapTimeoutTotal;
+ private final LongCounter otelDirectTimeoutTotal;
+
+ public TopicListMemoryLimiter(CollectorRegistry collectorRegistry, String prometheusPrefix,
+ Meter openTelemetryMeter,
+ long maxHeapMemory, int maxHeapQueueSize,
+ long heapTimeoutMillis, long maxDirectMemory, int maxDirectQueueSize,
+ long directTimeoutMillis) {
+ super(maxHeapMemory, maxHeapQueueSize, heapTimeoutMillis, maxDirectMemory, maxDirectQueueSize,
+ directTimeoutMillis);
+ this.collectorRegistry = collectorRegistry;
+
+ AsyncSemaphore heapMemoryLimiter = getLimiter(LimitType.HEAP_MEMORY);
+ AsyncSemaphore directMemoryLimiter = getLimiter(LimitType.DIRECT_MEMORY);
+
+ this.heapMemoryUsedBytes = register(Gauge.build(prometheusPrefix + "topic_list_heap_memory_used_bytes",
+ "Current heap memory used by topic listings")
+ .create()
+ .setChild(new Gauge.Child() {
+ @Override
+ public double get() {
+ return heapMemoryLimiter.getAcquiredPermits();
+ }
+ }));
+ this.otelHeapMemoryUsedGauge = openTelemetryMeter.gaugeBuilder("topic.list.heap.memory.used")
+ .setUnit("By")
+ .setDescription("Current heap memory used by topic listings")
+ .buildWithCallback(observableDoubleMeasurement -> {
+ observableDoubleMeasurement.record(heapMemoryLimiter.getAcquiredPermits());
+ });
+
+ this.heapMemoryLimitBytes = register(Gauge.build(prometheusPrefix + "topic_list_heap_memory_limit_bytes",
+ "Configured heap memory limit")
+ .create());
+ this.heapMemoryLimitBytes.set(maxHeapMemory);
+ this.otelHeapMemoryLimitGauge = openTelemetryMeter.gaugeBuilder("topic.list.heap.memory.limit")
+ .setUnit("By")
+ .setDescription("Configured heap memory limit")
+ .build();
+ this.otelHeapMemoryLimitGauge.set(maxHeapMemory);
+
+ this.directMemoryUsedBytes = register(Gauge.build(prometheusPrefix + "topic_list_direct_memory_used_bytes",
+ "Current direct memory used by topic listings")
+ .create()
+ .setChild(new Gauge.Child() {
+ @Override
+ public double get() {
+ return directMemoryLimiter.getAcquiredPermits();
+ }
+ }));
+ this.otelDirectMemoryUsedGauge = openTelemetryMeter.gaugeBuilder("topic.list.direct.memory.used")
+ .setUnit("By")
+ .setDescription("Current direct memory used by topic listings")
+ .buildWithCallback(observableDoubleMeasurement -> {
+ observableDoubleMeasurement.record(directMemoryLimiter.getAcquiredPermits());
+ });
+
+ this.directMemoryLimitBytes = register(Gauge.build(prometheusPrefix + "topic_list_direct_memory_limit_bytes",
+ "Configured direct memory limit")
+ .create());
+ this.directMemoryLimitBytes.set(maxDirectMemory);
+ this.otelDirectMemoryLimitGauge = openTelemetryMeter.gaugeBuilder("topic.list.direct.memory.limit")
+ .setUnit("By")
+ .setDescription("Configured direct memory limit")
+ .build();
+ this.otelDirectMemoryLimitGauge.set(maxDirectMemory);
+
+ this.heapQueueSize = register(Gauge.build(prometheusPrefix + "topic_list_heap_queue_size",
+ "Current heap memory limiter queue size")
+ .create()
+ .setChild(new Gauge.Child() {
+ @Override
+ public double get() {
+ return heapMemoryLimiter.getQueueSize();
+ }
+ }));
+ this.otelHeapQueueSize = openTelemetryMeter
+ .upDownCounterBuilder("topic.list.heap.queue.size")
+ .setDescription("Current heap memory limiter queue size")
+ .setUnit("1")
+ .buildWithCallback(observableLongMeasurement -> {
+ observableLongMeasurement.record(heapMemoryLimiter.getQueueSize());
+ });
+
+ this.heapQueueMaxSize = register(Gauge.build(prometheusPrefix + "topic_list_heap_queue_max_size",
+ "Maximum heap memory limiter queue size")
+ .create());
+ this.heapQueueMaxSize.set(maxHeapQueueSize);
+ LongGauge otelHeapQueueMaxSize = openTelemetryMeter
+ .gaugeBuilder("topic.list.heap.queue.max.size")
+ .setDescription("Maximum heap memory limiter queue size")
+ .setUnit("1")
+ .ofLongs()
+ .build();
+ otelHeapQueueMaxSize.set(maxHeapQueueSize);
+
+ this.directQueueSize = register(Gauge.build(prometheusPrefix + "topic_list_direct_queue_size",
+ "Current direct memory limiter queue size")
+ .create()
+ .setChild(new Gauge.Child() {
+ @Override
+ public double get() {
+ return directMemoryLimiter.getQueueSize();
+ }
+ }));
+ this.otelDirectQueueSize = openTelemetryMeter
+ .upDownCounterBuilder("topic.list.direct.queue.size")
+ .setDescription("Current direct memory limiter queue size")
+ .setUnit("1")
+ .buildWithCallback(observableLongMeasurement -> {
+ observableLongMeasurement.record(directMemoryLimiter.getQueueSize());
+ });
+
+ this.directQueueMaxSize = register(Gauge.build(prometheusPrefix + "topic_list_direct_queue_max_size",
+ "Maximum direct memory limiter queue size")
+ .create());
+ this.directQueueMaxSize.set(maxDirectQueueSize);
+ LongGauge otelDirectQueueMaxSize = openTelemetryMeter
+ .gaugeBuilder("topic.list.direct.queue.max.size")
+ .setDescription("Maximum direct memory limiter queue size")
+ .setUnit("1")
+ .ofLongs()
+ .build();
+ otelDirectQueueMaxSize.set(maxDirectQueueSize);
+
+ this.heapWaitTimeMs = register(Summary.build(prometheusPrefix + "topic_list_heap_wait_time_ms",
Review Comment:
I assume you are referencing to ["Summary is widely used, but not aggregatable across brokers / labels"](https://github.com/apache/pulsar/blob/master/pip/pip-264.md#summary-is-widely-used-but-not-aggregatable-across-brokers--labels). In this case, it's a Prometheus metric and we have been using summaries. I think that there's only very few locations where Prometheus histograms are used.
Just wondering if we should just keep on using a summary in this case since there's not really a need to aggregate this metric.
For the Open Telemetry metric `topic.list.heap.wait.time`, `io.opentelemetry.api.metrics.DoubleHistogram` is already used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2429832132
##########
pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/LookupProxyHandler.java:
##########
@@ -357,6 +357,59 @@ private void performGetTopicsOfNamespace(long clientRequestId,
});
}
+ private void internalPerformGetTopicsOfNamespace(long clientRequestId, String namespaceName, ClientCnx clientCnx,
+ ByteBuf command, long requestId) {
+ BooleanSupplier isPermitRequestCancelled = () -> !proxyConnection.ctx().channel().isActive();
+ maxTopicListInFlightLimiter.withAcquiredPermits(INITIAL_TOPIC_LIST_HEAP_PERMITS_SIZE,
+ AsyncDualMemoryLimiter.LimitType.HEAP_MEMORY, isPermitRequestCancelled, initialPermits -> {
+ return clientCnx.newGetTopicsOfNamespace(command, requestId).whenComplete((r, t) -> {
+ if (t != null) {
+ log.warn("[{}] Failed to get TopicsOfNamespace {}: {}", clientAddress, namespaceName,
+ t.getMessage());
+ writeAndFlush(Commands.newError(clientRequestId, getServerError(t), t.getMessage()));
+ } else {
+ long actualSize =
+ r.getNonPartitionedOrPartitionTopics().stream().mapToInt(String::length).sum();
Review Comment:
My assumption is that String are encoded as UTF-8 in the heap. There could be multi-byte characters which make the byte count larger than the character count. I've revisited the calculation to use byte count for UTF-8 encoding of characters.
I also added 32 byte overhead for each entry in the calculation.
The accuracy isn't that relevant in this case, since the limiter will introduce backpressure. The limit should be set to a low enough value so that actual resources don't get exhausted when the limit is reached.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2427144553
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/ServerCnx.java:
##########
@@ -2606,6 +2577,77 @@ protected void handleGetTopicsOfNamespace(CommandGetTopicsOfNamespace commandGet
}
}
+ private void internalHandleGetTopicsOfNamespace(String namespace, NamespaceName namespaceName, long requestId,
+ CommandGetTopicsOfNamespace.Mode mode,
+ Optional<String> topicsPattern, Optional<String> topicsHash,
+ Semaphore lookupSemaphore) {
+ BooleanSupplier isPermitRequestCancelled = () -> !ctx().channel().isActive();
+ maxTopicListInFlightLimiter.withAcquiredPermits(INITIAL_TOPIC_LIST_HEAP_PERMITS_SIZE,
+ AsyncDualMemoryLimiter.LimitType.HEAP_MEMORY, isPermitRequestCancelled, initialPermits -> {
+ return getBrokerService().pulsar().getNamespaceService().getListOfUserTopics(namespaceName, mode)
+ .thenAccept(topics -> {
+ long actualSize = topics.stream().mapToInt(String::length).sum();
+ maxTopicListInFlightLimiter.withUpdatedPermits(initialPermits, actualSize,
+ isPermitRequestCancelled, permits -> {
+ boolean filterTopics = false;
+ // filter system topic
+ List<String> filteredTopics = topics;
+
+ if (enableSubscriptionPatternEvaluation && topicsPattern.isPresent()) {
+ if (topicsPattern.get().length() <= maxSubscriptionPatternLength) {
+ filterTopics = true;
+ filteredTopics = TopicList.filterTopics(filteredTopics, topicsPattern.get(),
+ topicsPatternImplementation);
+ } else {
+ log.info("[{}] Subscription pattern provided [{}] was longer than "
+ + "maximum {}.", remoteAddress, topicsPattern.get(),
+ maxSubscriptionPatternLength);
+ }
+ }
+ String hash = TopicList.calculateHash(filteredTopics);
+ boolean hashUnchanged = topicsHash.isPresent() && topicsHash.get().equals(hash);
+ if (hashUnchanged) {
+ filteredTopics = Collections.emptyList();
+ }
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Received CommandGetTopicsOfNamespace for namespace "
+ + "[//{}] by {}, size:{}", remoteAddress, namespace,
+ requestId,
+ topics.size());
+ }
+ commandSender.sendGetTopicsOfNamespaceResponse(filteredTopics, hash, filterTopics,
+ !hashUnchanged, requestId, ex -> {
+ log.warn("[{}] Failed to acquire direct memory permits for "
+ + "GetTopicsOfNamespace: {}", remoteAddress, ex.getMessage());
+ commandSender.sendErrorResponse(requestId, ServerError.TooManyRequests,
+ "Cannot acquire permits for direct memory");
+ });
+ return CompletableFuture.completedFuture(null);
+ }, t -> {
+ log.warn("[{}] Failed to acquire heap memory permits for "
+ + "GetTopicsOfNamespace: {}", remoteAddress, t.getMessage());
+ writeAndFlush(Commands.newError(requestId, ServerError.TooManyRequests,
+ "Failed due to heap memory limit exceeded"));
+ return CompletableFuture.completedFuture(null);
+ });
+ }).whenComplete((__, ___) -> {
+ lookupSemaphore.release();
+ }).exceptionally(ex -> {
+ log.warn("[{}] Error GetTopicsOfNamespace for namespace [//{}] by {}", remoteAddress,
+ namespace, requestId);
+ commandSender.sendErrorResponse(requestId,
+ BrokerServiceException.getClientErrorCode(new ServerMetadataException(ex)),
+ ex.getMessage());
+ return null;
+ });
+ }, t -> {
+ log.warn("[{}] Failed to acquire initial heap memory permits for GetTopicsOfNamespace: {}",
+ remoteAddress, t.getMessage());
+ writeAndFlush(Commands.newError(requestId, ServerError.TooManyRequests,
+ "Failed due to heap memory limit exceeded"));
+ return CompletableFuture.completedFuture(null);
Review Comment:
yes, added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "poorbarcode (via GitHub)" <gi...@apache.org>.
poorbarcode commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2487256099
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/ServerCnx.java:
##########
@@ -2610,6 +2581,93 @@ protected void handleGetTopicsOfNamespace(CommandGetTopicsOfNamespace commandGet
}
}
+ private void internalHandleGetTopicsOfNamespace(String namespace, NamespaceName namespaceName, long requestId,
+ CommandGetTopicsOfNamespace.Mode mode,
+ Optional<String> topicsPattern, Optional<String> topicsHash,
+ Semaphore lookupSemaphore) {
+ BooleanSupplier isPermitRequestCancelled = () -> !ctx().channel().isActive();
+ TopicListSizeResultCache.ResultHolder
+ listSizeHolder = service.getTopicListSizeResultCache().getTopicListSize(namespaceName.toString(), mode);
+ listSizeHolder.getSizeAsync().thenAccept(initialSize -> {
+ maxTopicListInFlightLimiter.withAcquiredPermits(initialSize,
+ AsyncDualMemoryLimiter.LimitType.HEAP_MEMORY, isPermitRequestCancelled, initialPermits -> {
+ return getBrokerService().pulsar().getNamespaceService()
+ .getListOfUserTopics(namespaceName, mode)
+ .thenAccept(topics -> {
+ long actualSize = TopicListMemoryLimiter.estimateTopicListSize(topics);
+ listSizeHolder.updateSize(actualSize);
+ maxTopicListInFlightLimiter.withUpdatedPermits(initialPermits, actualSize,
Review Comment:
I just want to throw an improvement: Instead of updating and acquiring additional permits if the actual size is larger than estimated, we can skip requesting the additional permits and just mark it as "beyond the original value".
For example:
- max permits available: `100`
- estimated `100`
- acquire `100`
- get the real size `110`
- Apply directly without checking if the remaining permission is sufficient
- permits left: `-10`
---
- Benefit: An already applied 100 permits will not block subsequent requests
- Disadvantage: It has broken through the limitations
---
Anyway, I feel that the current implementation is also fine
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2421601390
##########
pulsar-broker-common/src/main/java/org/apache/pulsar/broker/topiclistlimit/TopicListMemoryLimiter.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.topiclistlimit;
+
+import io.opentelemetry.api.metrics.DoubleGauge;
+import io.opentelemetry.api.metrics.DoubleHistogram;
+import io.opentelemetry.api.metrics.LongCounter;
+import io.opentelemetry.api.metrics.LongGauge;
+import io.opentelemetry.api.metrics.Meter;
+import io.opentelemetry.api.metrics.ObservableDoubleGauge;
+import io.opentelemetry.api.metrics.ObservableLongUpDownCounter;
+import io.prometheus.client.Collector;
+import io.prometheus.client.CollectorRegistry;
+import io.prometheus.client.Counter;
+import io.prometheus.client.Gauge;
+import io.prometheus.client.Summary;
+import java.util.concurrent.TimeUnit;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.common.semaphore.AsyncDualMemoryLimiterImpl;
+import org.apache.pulsar.common.semaphore.AsyncSemaphore;
+
+/**
+ * Topic list memory limiter that exposes both Prometheus metrics and OpenTelemetry metrics.
+ */
+@Slf4j
+public class TopicListMemoryLimiter extends AsyncDualMemoryLimiterImpl {
+ private final CollectorRegistry collectorRegistry;
+ private final Gauge heapMemoryUsedBytes;
+ private final Gauge heapMemoryLimitBytes;
+ private final Gauge directMemoryUsedBytes;
+ private final Gauge directMemoryLimitBytes;
+ private final Gauge heapQueueSize;
+ private final Gauge heapQueueMaxSize;
+ private final Gauge directQueueSize;
+ private final Gauge directQueueMaxSize;
+ private final Summary heapWaitTimeMs;
+ private final Summary directWaitTimeMs;
+ private final Counter heapTimeoutTotal;
+ private final Counter directTimeoutTotal;
+ private final ObservableDoubleGauge otelHeapMemoryUsedGauge;
+ private final DoubleGauge otelHeapMemoryLimitGauge;
+ private final ObservableDoubleGauge otelDirectMemoryUsedGauge;
+ private final DoubleGauge otelDirectMemoryLimitGauge;
+ private final ObservableLongUpDownCounter otelHeapQueueSize;
+ private final ObservableLongUpDownCounter otelDirectQueueSize;
+ private final DoubleHistogram otelHeapWaitTime;
+ private final DoubleHistogram otelDirectWaitTime;
+ private final LongCounter otelHeapTimeoutTotal;
+ private final LongCounter otelDirectTimeoutTotal;
+
+ public TopicListMemoryLimiter(CollectorRegistry collectorRegistry, String prometheusPrefix,
+ Meter openTelemetryMeter,
+ long maxHeapMemory, int maxHeapQueueSize,
+ long heapTimeoutMillis, long maxDirectMemory, int maxDirectQueueSize,
+ long directTimeoutMillis) {
+ super(maxHeapMemory, maxHeapQueueSize, heapTimeoutMillis, maxDirectMemory, maxDirectQueueSize,
+ directTimeoutMillis);
+ this.collectorRegistry = collectorRegistry;
+
+ AsyncSemaphore heapMemoryLimiter = getLimiter(LimitType.HEAP_MEMORY);
+ AsyncSemaphore directMemoryLimiter = getLimiter(LimitType.DIRECT_MEMORY);
+
+ this.heapMemoryUsedBytes = register(Gauge.build(prometheusPrefix + "topic_list_heap_memory_used_bytes",
+ "Current heap memory used by topic listings")
+ .create()
+ .setChild(new Gauge.Child() {
+ @Override
+ public double get() {
+ return heapMemoryLimiter.getAcquiredPermits();
+ }
+ }));
+ this.otelHeapMemoryUsedGauge = openTelemetryMeter.gaugeBuilder("topic.list.heap.memory.used")
+ .setUnit("By")
+ .setDescription("Current heap memory used by topic listings")
+ .buildWithCallback(observableDoubleMeasurement -> {
+ observableDoubleMeasurement.record(heapMemoryLimiter.getAcquiredPermits());
+ });
+
+ this.heapMemoryLimitBytes = register(Gauge.build(prometheusPrefix + "topic_list_heap_memory_limit_bytes",
+ "Configured heap memory limit")
+ .create());
+ this.heapMemoryLimitBytes.set(maxHeapMemory);
+ this.otelHeapMemoryLimitGauge = openTelemetryMeter.gaugeBuilder("topic.list.heap.memory.limit")
+ .setUnit("By")
+ .setDescription("Configured heap memory limit")
+ .build();
+ this.otelHeapMemoryLimitGauge.set(maxHeapMemory);
+
+ this.directMemoryUsedBytes = register(Gauge.build(prometheusPrefix + "topic_list_direct_memory_used_bytes",
+ "Current direct memory used by topic listings")
+ .create()
+ .setChild(new Gauge.Child() {
+ @Override
+ public double get() {
+ return directMemoryLimiter.getAcquiredPermits();
+ }
+ }));
+ this.otelDirectMemoryUsedGauge = openTelemetryMeter.gaugeBuilder("topic.list.direct.memory.used")
+ .setUnit("By")
+ .setDescription("Current direct memory used by topic listings")
+ .buildWithCallback(observableDoubleMeasurement -> {
+ observableDoubleMeasurement.record(directMemoryLimiter.getAcquiredPermits());
+ });
+
+ this.directMemoryLimitBytes = register(Gauge.build(prometheusPrefix + "topic_list_direct_memory_limit_bytes",
+ "Configured direct memory limit")
+ .create());
+ this.directMemoryLimitBytes.set(maxDirectMemory);
+ this.otelDirectMemoryLimitGauge = openTelemetryMeter.gaugeBuilder("topic.list.direct.memory.limit")
+ .setUnit("By")
+ .setDescription("Configured direct memory limit")
+ .build();
+ this.otelDirectMemoryLimitGauge.set(maxHeapMemory);
Review Comment:
The direct memory limit gauge is being set with the heap memory value instead of the direct memory value. This should be `maxDirectMemory`.
```suggestion
this.otelDirectMemoryLimitGauge.set(maxDirectMemory);
```
##########
pulsar-common/src/test/java/org/apache/pulsar/common/semaphore/AsyncDualMemoryLimiterImplTest.java:
##########
@@ -0,0 +1,722 @@
+
Review Comment:
The file starts with an empty line before the license header. Remove the empty line to maintain consistency with the project's file format.
```suggestion
```
##########
pulsar-common/src/test/java/org/apache/pulsar/common/semaphore/AsyncDualMemoryLimiterUtilTest.java:
##########
@@ -0,0 +1,497 @@
+
Review Comment:
The file starts with an empty line before the license header. Remove the empty line to maintain consistency with the project's file format.
```suggestion
```
##########
pulsar-common/src/main/java/org/apache/pulsar/common/semaphore/AsyncDualMemoryLimiterUtil.java:
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.common.semaphore;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.channel.ChannelHandlerContext;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.BooleanSupplier;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import lombok.experimental.UtilityClass;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.common.api.proto.BaseCommand;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.semaphore.AsyncDualMemoryLimiter.AsyncDualMemoryLimiterPermit;
+
+@UtilityClass
+public class AsyncDualMemoryLimiterUtil {
+
+ public static <T> CompletableFuture<T> withPermitsFuture(
+ CompletableFuture<AsyncDualMemoryLimiterPermit>
+ permitsFuture,
+ Function<AsyncDualMemoryLimiterPermit,
+ CompletableFuture<T>> function,
+ Function<Throwable, CompletableFuture<T>>
+ permitAcquireErrorHandler,
+ Consumer<AsyncDualMemoryLimiterPermit> releaser) {
+ return permitsFuture
+ // combine the permits and error into a single pair so that it can be used in thenCompose
+ .handle((permits, permitAcquireError) ->
+ Pair.of(permits, permitAcquireError))
+ .thenCompose(permitsAndError -> {
+ if (permitsAndError.getRight() != null) {
+ // permits weren't acquired
+ return permitAcquireErrorHandler.apply(permitsAndError.getRight());
+ } else {
+ // permits were acquired
+ AsyncDualMemoryLimiterPermit permits = permitsAndError.getLeft();
+ try {
+ return function.apply(permits)
+ .whenComplete((__, ___) ->
+ // release the permits
+ releaser.accept(permits));
+ } catch (Throwable t) {
+ // release the permits if an exception occurs before the function returns
+ releaser.accept(permits);
+ throw t;
+ }
+ }
+ });
+ }
+
+ /**
+ * Acquire permits and write the command as the response to the channel.
+ * Releases the permits after the response has been written to the socket or the write fails.
+ *
+ * @param ctx the channel handler context.
+ * @param dualMemoryLimiter the memory limiter to acquire permits from.
+ * @param command the command to write to the channel.
+ * @return a future that completes when the command has been written to the channel's outbound buffer.
+ */
+ public static CompletableFuture<Void> acquireDirectMemoryPermitsAndWriteAndFlush(ChannelHandlerContext ctx,
+ AsyncDualMemoryLimiter
+ dualMemoryLimiter,
+ BooleanSupplier isCancelled,
+ BaseCommand command,
+ Consumer<Throwable>
+ permitAcquireErrorHandler
+ ) {
+ // Calculate serialized size before acquiring permits
+ int serializedSize = command.getSerializedSize();
+ // Acquire permits
+ return dualMemoryLimiter.acquire(serializedSize, AsyncDualMemoryLimiter.LimitType.DIRECT_MEMORY, isCancelled)
+ .whenComplete((permits, t) -> {
+ if (t != null) {
+ permitAcquireErrorHandler.accept(t);
+ return;
+ }
+ try {
+ // Serialize the response
+ ByteBuf outBuf = Commands.serializeWithPrecalculatedSerializedSize(command, serializedSize);
+ // Write the response
+ ctx.writeAndFlush(outBuf).addListener(future -> {
+ // Release permits after the response has been written to the socket
+ dualMemoryLimiter.release(permits);
+ });
+ } catch (Exception e) {
+ // Return permits if an exception occurs before writeAndFlush is called successfully
+ dualMemoryLimiter.release(permits);
+ throw e;
+ }
+ }).thenAccept(__ -> {
Review Comment:
The empty lambda body `thenAccept(__ -> {})` is unnecessary. Consider returning the CompletableFuture directly or using a more meaningful operation.
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2426525179
##########
org/apache/pulsar/common/util/memory/AsyncSemaphorePermitResult.java:
##########
Review Comment:
This file is obsolete, I'll remove this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2426525179
##########
org/apache/pulsar/common/util/memory/AsyncSemaphorePermitResult.java:
##########
Review Comment:
This file is obsolete, I'll remove this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2432248875
##########
pulsar-common/src/main/java/org/apache/pulsar/common/semaphore/AsyncSemaphoreImpl.java:
##########
@@ -0,0 +1,286 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.common.semaphore;
+
+import io.netty.util.concurrent.DefaultThreadFactory;
+import java.util.Queue;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicLongFieldUpdater;
+import java.util.function.BooleanSupplier;
+import java.util.function.LongConsumer;
+import org.apache.pulsar.common.util.Runnables;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of AsyncSemaphore with timeout and queue size limits.
+ */
+public class AsyncSemaphoreImpl implements AsyncSemaphore, AutoCloseable {
+ private static final Logger log = LoggerFactory.getLogger(AsyncSemaphoreImpl.class);
+
+ private final AtomicLong availablePermits;
+ private final Queue<PendingRequest> queue;
+ private final long maxPermits;
+ private final long timeoutMillis;
+ private final ScheduledExecutorService executor;
+ private final boolean shutdownExecutor;
+ private final LongConsumer queueLatencyRecorder;
+ private final AtomicBoolean closed = new AtomicBoolean(false);
+ private final Runnable processQueueRunnable = Runnables.catchingAndLoggingThrowables(this::internalProcessQueue);
+
+ public AsyncSemaphoreImpl(long maxPermits, int maxQueueSize, long timeoutMillis) {
+ this(maxPermits, maxQueueSize, timeoutMillis, createExecutor(), true, null);
+ }
+
+ public AsyncSemaphoreImpl(long maxPermits, int maxQueueSize, long timeoutMillis,
+ ScheduledExecutorService executor, LongConsumer queueLatencyRecorder) {
+ this(maxPermits, maxQueueSize, timeoutMillis, executor, false, queueLatencyRecorder);
+ }
+
+ AsyncSemaphoreImpl(long maxPermits, int maxQueueSize, long timeoutMillis, ScheduledExecutorService executor,
+ boolean shutdownExecutor, LongConsumer queueLatencyRecorder) {
+ this.availablePermits = new AtomicLong(maxPermits);
+ this.maxPermits = maxPermits;
+ this.queue = new ArrayBlockingQueue<>(maxQueueSize);
+ this.timeoutMillis = timeoutMillis;
+ this.executor = executor;
+ this.shutdownExecutor = shutdownExecutor;
+ this.queueLatencyRecorder = queueLatencyRecorder;
+ }
+
+ private static ScheduledExecutorService createExecutor() {
+ return Executors.newSingleThreadScheduledExecutor(
+ new DefaultThreadFactory("async-semaphore-executor"));
+ }
+
+ @Override
+ public CompletableFuture<AsyncSemaphorePermit> acquire(long permits, BooleanSupplier isCancelled) {
+ return internalAcquire(permits, permits, isCancelled);
+ }
+
+ private CompletableFuture<AsyncSemaphorePermit> internalAcquire(long permits, long acquirePermits,
+ BooleanSupplier isCancelled) {
+ if (permits < 0) {
+ throw new IllegalArgumentException("Invalid permits value: " + permits);
+ }
+
+ CompletableFuture<AsyncSemaphorePermit> future = new CompletableFuture<>();
+
+ if (closed.get()) {
+ future.completeExceptionally(new PermitAcquireAlreadyClosedException("Semaphore is closed"));
+ return future;
+ }
+
+ PendingRequest request = new PendingRequest(permits, acquirePermits, future, isCancelled);
+ if (!queue.offer(request)) {
+ future.completeExceptionally(new PermitAcquireQueueFullException(
+ "Semaphore queue is full"));
+ return future;
+ }
+ // Schedule timeout
+ ScheduledFuture<?> timeoutTask = executor.schedule(() -> {
+ if (!request.future.isDone() && queue.remove(request)) {
+ // timeout is recorded with Long.MAX_VALUE as the age
+ recordQueueLatency(Long.MAX_VALUE);
+ // also record the time in the queue
+ recordQueueLatency(request.getAgeNanos());
+ future.completeExceptionally(new PermitAcquireTimeoutException(
+ "Permit acquisition timed out"));
+ // the next request might have smaller permits and that might be processed
+ processQueue();
+ }
+ }, timeoutMillis, TimeUnit.MILLISECONDS);
+ request.setTimeoutTask(timeoutTask);
+
+ processQueue();
+ return future;
+ }
+
+ private void recordQueueLatency(long ageNanos) {
+ if (queueLatencyRecorder != null) {
+ queueLatencyRecorder.accept(ageNanos);
+ }
+ }
+
+ @Override
+ public CompletableFuture<AsyncSemaphorePermit> update(AsyncSemaphorePermit permit, long newPermits,
+ BooleanSupplier isCancelled) {
+ if (newPermits < 0) {
+ throw new IllegalArgumentException("Invalid permits value: " + newPermits);
+ }
+ long oldPermits = permit.getPermits();
+ long additionalPermits = newPermits - oldPermits;
+ // mark the old permits as released without adding the permits to availablePermits
+ castToImplementation(permit).releasePermits();
+ if (additionalPermits > 0) {
+ return internalAcquire(newPermits, additionalPermits, isCancelled);
+ } else {
+ // new permits are less than the old ones, so we return the difference
+ availablePermits.addAndGet(-additionalPermits);
+ processQueue();
+ // return the new permits immediately
+ return CompletableFuture.completedFuture(new SemaphorePermit(newPermits));
+ }
+ }
+
+ @Override
+ public void release(AsyncSemaphorePermit permit) {
+ availablePermits.addAndGet(castToImplementation(permit).releasePermits());
+ processQueue();
+ }
+
+ @Override
+ public long getAvailablePermits() {
+ return availablePermits.get();
+ }
+
+ @Override
+ public long getAcquiredPermits() {
+ return maxPermits - availablePermits.get();
+ }
+
+ @Override
+ public int getQueueSize() {
+ return queue.size();
+ }
+
+ private SemaphorePermit castToImplementation(AsyncSemaphorePermit permit) {
+ if (permit instanceof SemaphorePermit semaphorePermit) {
+ return semaphorePermit;
+ } else {
+ throw new IllegalArgumentException("Invalid permit type");
+ }
+ }
+
+ private void processQueue() {
+ if (closed.get()) {
+ return;
+ }
+ executor.execute(processQueueRunnable);
+ }
+
+ private void internalProcessQueue() {
Review Comment:
This method is always run in a single thread by a single threaded executor. I guess it would need to be documented that the executor needs to be single threaded.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2433444140
##########
pulsar-common/src/main/java/org/apache/pulsar/common/semaphore/AsyncSemaphoreImpl.java:
##########
@@ -0,0 +1,286 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.common.semaphore;
+
+import io.netty.util.concurrent.DefaultThreadFactory;
+import java.util.Queue;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicLongFieldUpdater;
+import java.util.function.BooleanSupplier;
+import java.util.function.LongConsumer;
+import org.apache.pulsar.common.util.Runnables;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of AsyncSemaphore with timeout and queue size limits.
+ */
+public class AsyncSemaphoreImpl implements AsyncSemaphore, AutoCloseable {
+ private static final Logger log = LoggerFactory.getLogger(AsyncSemaphoreImpl.class);
+
+ private final AtomicLong availablePermits;
+ private final Queue<PendingRequest> queue;
+ private final long maxPermits;
+ private final long timeoutMillis;
+ private final ScheduledExecutorService executor;
+ private final boolean shutdownExecutor;
+ private final LongConsumer queueLatencyRecorder;
+ private final AtomicBoolean closed = new AtomicBoolean(false);
+ private final Runnable processQueueRunnable = Runnables.catchingAndLoggingThrowables(this::internalProcessQueue);
+
+ public AsyncSemaphoreImpl(long maxPermits, int maxQueueSize, long timeoutMillis) {
+ this(maxPermits, maxQueueSize, timeoutMillis, createExecutor(), true, null);
+ }
+
+ public AsyncSemaphoreImpl(long maxPermits, int maxQueueSize, long timeoutMillis,
+ ScheduledExecutorService executor, LongConsumer queueLatencyRecorder) {
+ this(maxPermits, maxQueueSize, timeoutMillis, executor, false, queueLatencyRecorder);
+ }
+
+ AsyncSemaphoreImpl(long maxPermits, int maxQueueSize, long timeoutMillis, ScheduledExecutorService executor,
+ boolean shutdownExecutor, LongConsumer queueLatencyRecorder) {
+ this.availablePermits = new AtomicLong(maxPermits);
+ this.maxPermits = maxPermits;
+ this.queue = new ArrayBlockingQueue<>(maxQueueSize);
+ this.timeoutMillis = timeoutMillis;
+ this.executor = executor;
+ this.shutdownExecutor = shutdownExecutor;
+ this.queueLatencyRecorder = queueLatencyRecorder;
+ }
+
+ private static ScheduledExecutorService createExecutor() {
+ return Executors.newSingleThreadScheduledExecutor(
+ new DefaultThreadFactory("async-semaphore-executor"));
+ }
+
+ @Override
+ public CompletableFuture<AsyncSemaphorePermit> acquire(long permits, BooleanSupplier isCancelled) {
+ return internalAcquire(permits, permits, isCancelled);
+ }
+
+ private CompletableFuture<AsyncSemaphorePermit> internalAcquire(long permits, long acquirePermits,
+ BooleanSupplier isCancelled) {
+ if (permits < 0) {
+ throw new IllegalArgumentException("Invalid permits value: " + permits);
+ }
+
+ CompletableFuture<AsyncSemaphorePermit> future = new CompletableFuture<>();
+
+ if (closed.get()) {
+ future.completeExceptionally(new PermitAcquireAlreadyClosedException("Semaphore is closed"));
+ return future;
+ }
+
+ PendingRequest request = new PendingRequest(permits, acquirePermits, future, isCancelled);
+ if (!queue.offer(request)) {
+ future.completeExceptionally(new PermitAcquireQueueFullException(
+ "Semaphore queue is full"));
+ return future;
+ }
+ // Schedule timeout
+ ScheduledFuture<?> timeoutTask = executor.schedule(() -> {
+ if (!request.future.isDone() && queue.remove(request)) {
+ // timeout is recorded with Long.MAX_VALUE as the age
+ recordQueueLatency(Long.MAX_VALUE);
+ // also record the time in the queue
+ recordQueueLatency(request.getAgeNanos());
+ future.completeExceptionally(new PermitAcquireTimeoutException(
+ "Permit acquisition timed out"));
+ // the next request might have smaller permits and that might be processed
+ processQueue();
+ }
+ }, timeoutMillis, TimeUnit.MILLISECONDS);
+ request.setTimeoutTask(timeoutTask);
+
+ processQueue();
+ return future;
+ }
+
+ private void recordQueueLatency(long ageNanos) {
+ if (queueLatencyRecorder != null) {
+ queueLatencyRecorder.accept(ageNanos);
+ }
+ }
+
+ @Override
+ public CompletableFuture<AsyncSemaphorePermit> update(AsyncSemaphorePermit permit, long newPermits,
+ BooleanSupplier isCancelled) {
+ if (newPermits < 0) {
+ throw new IllegalArgumentException("Invalid permits value: " + newPermits);
+ }
+ long oldPermits = permit.getPermits();
+ long additionalPermits = newPermits - oldPermits;
+ // mark the old permits as released without adding the permits to availablePermits
+ castToImplementation(permit).releasePermits();
+ if (additionalPermits > 0) {
+ return internalAcquire(newPermits, additionalPermits, isCancelled);
+ } else {
+ // new permits are less than the old ones, so we return the difference
+ availablePermits.addAndGet(-additionalPermits);
+ processQueue();
+ // return the new permits immediately
+ return CompletableFuture.completedFuture(new SemaphorePermit(newPermits));
+ }
+ }
+
+ @Override
+ public void release(AsyncSemaphorePermit permit) {
+ availablePermits.addAndGet(castToImplementation(permit).releasePermits());
+ processQueue();
+ }
+
+ @Override
+ public long getAvailablePermits() {
+ return availablePermits.get();
+ }
+
+ @Override
+ public long getAcquiredPermits() {
+ return maxPermits - availablePermits.get();
+ }
+
+ @Override
+ public int getQueueSize() {
+ return queue.size();
+ }
+
+ private SemaphorePermit castToImplementation(AsyncSemaphorePermit permit) {
+ if (permit instanceof SemaphorePermit semaphorePermit) {
+ return semaphorePermit;
+ } else {
+ throw new IllegalArgumentException("Invalid permit type");
+ }
+ }
+
+ private void processQueue() {
+ if (closed.get()) {
+ return;
+ }
+ executor.execute(processQueueRunnable);
+ }
+
+ private void internalProcessQueue() {
Review Comment:
I've documented the requirement of a single threaded executor and made the method synchronized to avoid any future confusion in maintaining the code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "poorbarcode (via GitHub)" <gi...@apache.org>.
poorbarcode commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2476677248
##########
pulsar-broker-common/src/main/java/org/apache/pulsar/broker/topiclistlimit/TopicListSizeResultCache.java:
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.topiclistlimit;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReference;
+import org.apache.pulsar.common.api.proto.CommandGetTopicsOfNamespace;
+
+/**
+ * A cache for storing and managing topic list size estimates in namespaces.
+ * This class provides functionality to:
+ * - Cache and retrieve topic list size estimates for different namespaces and modes
+ * - Handle concurrent requests for topic list sizes efficiently
+ * - Maintain and update size estimates based on actual topic list sizes
+ * - Prevent thundering herd problems when multiple concurrent requests for a namespace are made without
+ * a previous size estimate
+ * The cache uses namespace name and topic list mode (PERSISTENT/NON_PERSISTENT/ALL) as keys
+ * and maintains size estimates that are refined with actual usage.
+ */
+public class TopicListSizeResultCache {
+ // 10kB initial estimate for topic list heap size
+ private static final long INITIAL_TOPIC_LIST_HEAP_SIZE = 10 * 1024;
+
+ private Cache<CacheKey, ResultHolder> topicListSizeCache = Caffeine.newBuilder()
+ .maximumSize(10000)
+ .build();
+
+ record CacheKey(String namespaceName, CommandGetTopicsOfNamespace.Mode mode) {
+ }
+
+ /**
+ * Holds the topic list size estimate and future for the topic list size.
+ * The size is returned by calling {@link #getSizeAsync()} method which is asynchronous.
+ * The estimate is updated by calling {@link #updateSize(long)} method.
+ */
+ public static class ResultHolder {
+ private final AtomicReference<CompletableFuture<Long>> topicListSizeFuture =
Review Comment:
Okay, the first request will get a constant value, and the second will wait for the first request to be finished if it has not completed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2490378365
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/ServerCnx.java:
##########
@@ -2610,6 +2581,93 @@ protected void handleGetTopicsOfNamespace(CommandGetTopicsOfNamespace commandGet
}
}
+ private void internalHandleGetTopicsOfNamespace(String namespace, NamespaceName namespaceName, long requestId,
+ CommandGetTopicsOfNamespace.Mode mode,
+ Optional<String> topicsPattern, Optional<String> topicsHash,
+ Semaphore lookupSemaphore) {
+ BooleanSupplier isPermitRequestCancelled = () -> !ctx().channel().isActive();
+ TopicListSizeResultCache.ResultHolder
+ listSizeHolder = service.getTopicListSizeResultCache().getTopicListSize(namespaceName.toString(), mode);
+ listSizeHolder.getSizeAsync().thenAccept(initialSize -> {
+ maxTopicListInFlightLimiter.withAcquiredPermits(initialSize,
+ AsyncDualMemoryLimiter.LimitType.HEAP_MEMORY, isPermitRequestCancelled, initialPermits -> {
+ return getBrokerService().pulsar().getNamespaceService()
+ .getListOfUserTopics(namespaceName, mode)
+ .thenAccept(topics -> {
+ long actualSize = TopicListMemoryLimiter.estimateTopicListSize(topics);
+ listSizeHolder.updateSize(actualSize);
+ maxTopicListInFlightLimiter.withUpdatedPermits(initialPermits, actualSize,
Review Comment:
> Anyway, I feel that the current implementation is also fine
Ok, I'll keep the current implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "poorbarcode (via GitHub)" <gi...@apache.org>.
poorbarcode commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2445395653
##########
pulsar-broker-common/src/main/java/org/apache/pulsar/broker/topiclistlimit/TopicListMemoryLimiter.java:
##########
@@ -0,0 +1,312 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.topiclistlimit;
+
+import io.netty.buffer.ByteBufUtil;
+import io.opentelemetry.api.metrics.DoubleGauge;
+import io.opentelemetry.api.metrics.DoubleHistogram;
+import io.opentelemetry.api.metrics.LongCounter;
+import io.opentelemetry.api.metrics.LongGauge;
+import io.opentelemetry.api.metrics.Meter;
+import io.opentelemetry.api.metrics.ObservableDoubleGauge;
+import io.opentelemetry.api.metrics.ObservableLongUpDownCounter;
+import io.prometheus.client.Collector;
+import io.prometheus.client.CollectorRegistry;
+import io.prometheus.client.Counter;
+import io.prometheus.client.Gauge;
+import io.prometheus.client.Summary;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.common.semaphore.AsyncDualMemoryLimiterImpl;
+import org.apache.pulsar.common.semaphore.AsyncSemaphore;
+
+/**
+ * Topic list memory limiter that exposes both Prometheus metrics and OpenTelemetry metrics.
+ */
+@Slf4j
+public class TopicListMemoryLimiter extends AsyncDualMemoryLimiterImpl {
Review Comment:
Do we need to indicate in this class name that this is a class related to metrics?
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/TopicListService.java:
##########
@@ -101,6 +112,7 @@ public void accept(String topicName, NotificationType notificationType) {
private final boolean enableSubscriptionPatternEvaluation;
private final int maxSubscriptionPatternLength;
private final ConcurrentLongHashMap<CompletableFuture<TopicListWatcher>> watchers;
+ private final Backoff retryBackoff;
public TopicListService(PulsarService pulsar, ServerCnx connection,
Review Comment:
This function is a notification for the client, to let the client know which topic was changed; it should not be limited by the new feature. the change https://github.com/apache/pulsar/pull/24833/files#diff-5cb94e51fda5445d6fc829fda24eeccda65bf166643a7bf6957fa195516bca28L363-R392 is also related
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/NamespacesBase.java:
##########
@@ -165,8 +173,66 @@ protected CompletableFuture<Void> internalCreateNamespace(Policies policies) {
.thenAccept(__ -> log.info("[{}] Created namespace {}", clientAppId(), namespaceName));
}
- protected CompletableFuture<List<String>> internalGetListOfTopics(Policies policies,
+ protected CompletableFuture<List<String>> internalGetListOfTopics(AsyncResponse response, Policies policies,
CommandGetTopicsOfNamespace.Mode mode) {
+ // Use maxTopicListInFlightLimiter to limit inflight get topic listing responses
+ // to avoid OOME caused by a lot of clients using HTTP service lookups to list topics
+ AsyncDualMemoryLimiterImpl maxTopicListInFlightLimiter =
+ pulsar().getBrokerService().getMaxTopicListInFlightLimiter();
+ TopicListSizeResultCache.ResultHolder listSizeHolder = pulsar().getBrokerService().getTopicListSizeResultCache()
+ .getTopicListSize(namespaceName.toString(), mode);
+ // setup the permit cancellation function
+ AtomicBoolean permitRequestCancelled = new AtomicBoolean(false);
+ BooleanSupplier isPermitRequestCancelled = permitRequestCancelled::get;
+ // add callback that releases permits when the response completes
+ AtomicReference<AsyncDualMemoryLimiter.AsyncDualMemoryLimiterPermit> initialPermitsRef =
+ new AtomicReference<>();
+ AtomicReference<AsyncDualMemoryLimiter.AsyncDualMemoryLimiterPermit> permitsRef = new AtomicReference<>();
+ response.register(new CompletionCallback() {
+ @Override
+ public void onComplete(Throwable throwable) {
+ if (throwable != null) {
+ // for failed request
+ // handle resetting the TopicListSizeResultCache.ResultHolder
+ listSizeHolder.resetIfInitializing();
+ // cancel any pending permit request
+ permitRequestCancelled.set(true);
+ }
+ AsyncDualMemoryLimiter.AsyncDualMemoryLimiterPermit initialPermit = initialPermitsRef.get();
+ if (initialPermit != null) {
+ maxTopicListInFlightLimiter.release(initialPermit);
+ }
+ AsyncDualMemoryLimiter.AsyncDualMemoryLimiterPermit permits = permitsRef.get();
+ if (permits != null) {
+ maxTopicListInFlightLimiter.release(permits);
+ }
+ }
+ });
+ return listSizeHolder.getSizeAsync().thenCompose(initialSize -> maxTopicListInFlightLimiter.acquire(initialSize,
+ AsyncDualMemoryLimiter.LimitType.HEAP_MEMORY, isPermitRequestCancelled).exceptionally(t -> {
+ throw new CompletionException(
+ new RestException(Status.TOO_MANY_REQUESTS, "Failed due to heap memory limit exceeded"));
+ }).thenCompose(initialPermits -> {
+ initialPermitsRef.set(initialPermits);
+ // perform the actual get list of topics operation
+ return doInternalGetListOfTopics(policies, mode).thenCompose(topicList -> {
+ long actualSize = TopicListMemoryLimiter.estimateTopicListSize(topicList);
+ listSizeHolder.updateSize(actualSize);
+ return maxTopicListInFlightLimiter.update(initialPermits, actualSize, isPermitRequestCancelled)
Review Comment:
The same comment as https://github.com/apache/pulsar/pull/24833#discussion_r2444987116
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2500577397
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/TopicListService.java:
##########
@@ -101,6 +112,7 @@ public void accept(String topicName, NotificationType notificationType) {
private final boolean enableSubscriptionPatternEvaluation;
private final int maxSubscriptionPatternLength;
private final ConcurrentLongHashMap<CompletableFuture<TopicListWatcher>> watchers;
+ private final Backoff retryBackoff;
public TopicListService(PulsarService pulsar, ServerCnx connection,
Review Comment:
@poorbarcode I've addressed the required changes. Topic list watchers will work fine with this solution. In case of running out of permits, there will be infinite retries (until the watcher is closed by closing it's connection). Backpressure is about slowing down responses in this case to reduce the load on the system. I'll continue to add a test case where the problems of the existing solution would be demonstrated in the similar way as there's org.apache.pulsar.client.api.PatternConsumerBackPressureMultipleConsumersTest, but using actual client consumers that use topic list watchers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2506854677
##########
pulsar-broker-common/src/main/java/org/apache/pulsar/broker/topiclistlimit/TopicListMemoryLimiter.java:
##########
@@ -0,0 +1,312 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.topiclistlimit;
+
+import io.netty.buffer.ByteBufUtil;
+import io.opentelemetry.api.metrics.DoubleGauge;
+import io.opentelemetry.api.metrics.DoubleHistogram;
+import io.opentelemetry.api.metrics.LongCounter;
+import io.opentelemetry.api.metrics.LongGauge;
+import io.opentelemetry.api.metrics.Meter;
+import io.opentelemetry.api.metrics.ObservableDoubleGauge;
+import io.opentelemetry.api.metrics.ObservableLongUpDownCounter;
+import io.prometheus.client.Collector;
+import io.prometheus.client.CollectorRegistry;
+import io.prometheus.client.Counter;
+import io.prometheus.client.Gauge;
+import io.prometheus.client.Summary;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.common.semaphore.AsyncDualMemoryLimiterImpl;
+import org.apache.pulsar.common.semaphore.AsyncSemaphore;
+
+/**
+ * Topic list memory limiter that exposes both Prometheus metrics and OpenTelemetry metrics.
+ */
+@Slf4j
+public class TopicListMemoryLimiter extends AsyncDualMemoryLimiterImpl {
+ private final CollectorRegistry collectorRegistry;
+ private final Gauge heapMemoryUsedBytes;
+ private final Gauge heapMemoryLimitBytes;
+ private final Gauge directMemoryUsedBytes;
+ private final Gauge directMemoryLimitBytes;
+ private final Gauge heapQueueSize;
+ private final Gauge heapQueueMaxSize;
+ private final Gauge directQueueSize;
+ private final Gauge directQueueMaxSize;
+ private final Summary heapWaitTimeMs;
+ private final Summary directWaitTimeMs;
+ private final Counter heapTimeoutTotal;
+ private final Counter directTimeoutTotal;
+ private final ObservableDoubleGauge otelHeapMemoryUsedGauge;
+ private final DoubleGauge otelHeapMemoryLimitGauge;
+ private final ObservableDoubleGauge otelDirectMemoryUsedGauge;
+ private final DoubleGauge otelDirectMemoryLimitGauge;
+ private final ObservableLongUpDownCounter otelHeapQueueSize;
+ private final ObservableLongUpDownCounter otelDirectQueueSize;
+ private final DoubleHistogram otelHeapWaitTime;
+ private final DoubleHistogram otelDirectWaitTime;
+ private final LongCounter otelHeapTimeoutTotal;
+ private final LongCounter otelDirectTimeoutTotal;
+
+ public TopicListMemoryLimiter(CollectorRegistry collectorRegistry, String prometheusPrefix,
+ Meter openTelemetryMeter,
+ long maxHeapMemory, int maxHeapQueueSize,
+ long heapTimeoutMillis, long maxDirectMemory, int maxDirectQueueSize,
+ long directTimeoutMillis) {
+ super(maxHeapMemory, maxHeapQueueSize, heapTimeoutMillis, maxDirectMemory, maxDirectQueueSize,
+ directTimeoutMillis);
+ this.collectorRegistry = collectorRegistry;
+
+ AsyncSemaphore heapMemoryLimiter = getLimiter(LimitType.HEAP_MEMORY);
+ AsyncSemaphore directMemoryLimiter = getLimiter(LimitType.DIRECT_MEMORY);
+
+ this.heapMemoryUsedBytes = register(Gauge.build(prometheusPrefix + "topic_list_heap_memory_used_bytes",
+ "Current heap memory used by topic listings")
+ .create()
+ .setChild(new Gauge.Child() {
+ @Override
+ public double get() {
+ return heapMemoryLimiter.getAcquiredPermits();
+ }
+ }));
+ this.otelHeapMemoryUsedGauge = openTelemetryMeter.gaugeBuilder("topic.list.heap.memory.used")
+ .setUnit("By")
+ .setDescription("Current heap memory used by topic listings")
+ .buildWithCallback(observableDoubleMeasurement -> {
+ observableDoubleMeasurement.record(heapMemoryLimiter.getAcquiredPermits());
+ });
+
+ this.heapMemoryLimitBytes = register(Gauge.build(prometheusPrefix + "topic_list_heap_memory_limit_bytes",
+ "Configured heap memory limit")
+ .create());
+ this.heapMemoryLimitBytes.set(maxHeapMemory);
+ this.otelHeapMemoryLimitGauge = openTelemetryMeter.gaugeBuilder("topic.list.heap.memory.limit")
+ .setUnit("By")
+ .setDescription("Configured heap memory limit")
+ .build();
+ this.otelHeapMemoryLimitGauge.set(maxHeapMemory);
+
+ this.directMemoryUsedBytes = register(Gauge.build(prometheusPrefix + "topic_list_direct_memory_used_bytes",
+ "Current direct memory used by topic listings")
+ .create()
+ .setChild(new Gauge.Child() {
+ @Override
+ public double get() {
+ return directMemoryLimiter.getAcquiredPermits();
+ }
+ }));
+ this.otelDirectMemoryUsedGauge = openTelemetryMeter.gaugeBuilder("topic.list.direct.memory.used")
+ .setUnit("By")
+ .setDescription("Current direct memory used by topic listings")
+ .buildWithCallback(observableDoubleMeasurement -> {
+ observableDoubleMeasurement.record(directMemoryLimiter.getAcquiredPermits());
+ });
+
+ this.directMemoryLimitBytes = register(Gauge.build(prometheusPrefix + "topic_list_direct_memory_limit_bytes",
+ "Configured direct memory limit")
+ .create());
+ this.directMemoryLimitBytes.set(maxDirectMemory);
+ this.otelDirectMemoryLimitGauge = openTelemetryMeter.gaugeBuilder("topic.list.direct.memory.limit")
+ .setUnit("By")
+ .setDescription("Configured direct memory limit")
+ .build();
+ this.otelDirectMemoryLimitGauge.set(maxDirectMemory);
+
+ this.heapQueueSize = register(Gauge.build(prometheusPrefix + "topic_list_heap_queue_size",
+ "Current heap memory limiter queue size")
+ .create()
+ .setChild(new Gauge.Child() {
+ @Override
+ public double get() {
+ return heapMemoryLimiter.getQueueSize();
+ }
+ }));
+ this.otelHeapQueueSize = openTelemetryMeter
+ .upDownCounterBuilder("topic.list.heap.queue.size")
+ .setDescription("Current heap memory limiter queue size")
+ .setUnit("1")
+ .buildWithCallback(observableLongMeasurement -> {
+ observableLongMeasurement.record(heapMemoryLimiter.getQueueSize());
+ });
+
+ this.heapQueueMaxSize = register(Gauge.build(prometheusPrefix + "topic_list_heap_queue_max_size",
+ "Maximum heap memory limiter queue size")
+ .create());
+ this.heapQueueMaxSize.set(maxHeapQueueSize);
+ LongGauge otelHeapQueueMaxSize = openTelemetryMeter
+ .gaugeBuilder("topic.list.heap.queue.max.size")
+ .setDescription("Maximum heap memory limiter queue size")
+ .setUnit("1")
+ .ofLongs()
+ .build();
+ otelHeapQueueMaxSize.set(maxHeapQueueSize);
+
+ this.directQueueSize = register(Gauge.build(prometheusPrefix + "topic_list_direct_queue_size",
+ "Current direct memory limiter queue size")
+ .create()
+ .setChild(new Gauge.Child() {
+ @Override
+ public double get() {
+ return directMemoryLimiter.getQueueSize();
+ }
+ }));
+ this.otelDirectQueueSize = openTelemetryMeter
+ .upDownCounterBuilder("topic.list.direct.queue.size")
+ .setDescription("Current direct memory limiter queue size")
+ .setUnit("1")
+ .buildWithCallback(observableLongMeasurement -> {
+ observableLongMeasurement.record(directMemoryLimiter.getQueueSize());
+ });
+
+ this.directQueueMaxSize = register(Gauge.build(prometheusPrefix + "topic_list_direct_queue_max_size",
+ "Maximum direct memory limiter queue size")
+ .create());
+ this.directQueueMaxSize.set(maxDirectQueueSize);
+ LongGauge otelDirectQueueMaxSize = openTelemetryMeter
+ .gaugeBuilder("topic.list.direct.queue.max.size")
+ .setDescription("Maximum direct memory limiter queue size")
+ .setUnit("1")
+ .ofLongs()
+ .build();
+ otelDirectQueueMaxSize.set(maxDirectQueueSize);
+
+ this.heapWaitTimeMs = register(Summary.build(prometheusPrefix + "topic_list_heap_wait_time_ms",
+ "Wait time for heap memory permits")
+ .quantile(0.50, 0.01)
+ .quantile(0.95, 0.01)
+ .quantile(0.99, 0.01)
+ .quantile(1, 0.01)
+ .create());
+ this.otelHeapWaitTime = openTelemetryMeter.histogramBuilder("topic.list.heap.wait.time")
+ .setUnit("s")
+ .setDescription("Wait time for heap memory permits")
+ .build();
+
+ this.directWaitTimeMs = register(Summary.build(prometheusPrefix + "topic_list_direct_wait_time_ms",
+ "Wait time for direct memory permits")
+ .quantile(0.50, 0.01)
+ .quantile(0.95, 0.01)
+ .quantile(0.99, 0.01)
+ .quantile(1, 0.01)
+ .create());
+ this.otelDirectWaitTime = openTelemetryMeter.histogramBuilder("topic.list.direct.wait.time")
+ .setUnit("s")
+ .setDescription("Wait time for direct memory permits")
+ .build();
+
+ this.heapTimeoutTotal = register(Counter.build(prometheusPrefix + "topic_list_heap_timeout_total",
+ "Total heap memory permit timeouts")
+ .create());
+ this.otelHeapTimeoutTotal = openTelemetryMeter.counterBuilder("topic.list.heap.timeout.total")
+ .setDescription("Total heap memory permit timeouts")
+ .setUnit("1")
+ .build();
+
+ this.directTimeoutTotal = register(Counter.build(prometheusPrefix + "topic_list_direct_timeout_total",
+ "Total direct memory permit timeouts")
+ .create());
+ this.otelDirectTimeoutTotal = openTelemetryMeter.counterBuilder("topic.list.direct.timeout.total")
+ .setDescription("Total direct memory permit timeouts")
+ .setUnit("1")
+ .build();
+ }
+
+ private <T extends Collector> T register(T collector) {
+ try {
+ collectorRegistry.register(collector);
+ } catch (Exception e) {
+ // ignore exception when registering a collector that is already registered
+ if (log.isDebugEnabled()) {
+ log.debug("Failed to register Prometheus collector {}", collector, e);
+ }
+ }
+ return collector;
+ }
+
+ private void unregister(Collector collector) {
+ try {
+ collectorRegistry.unregister(collector);
+ } catch (Exception e) {
+ // ignore exception when unregistering a collector that is not registered
+ if (log.isDebugEnabled()) {
+ log.debug("Failed to unregister Prometheus collector {}", collector, e);
+ }
+ }
+ }
+
+
+ @Override
+ protected void recordHeapWaitTime(long waitTimeNanos) {
+ if (waitTimeNanos == Long.MAX_VALUE) {
+ heapTimeoutTotal.inc();
+ otelHeapTimeoutTotal.add(1);
+ } else {
+ heapWaitTimeMs.observe(TimeUnit.NANOSECONDS.toMillis(waitTimeNanos));
+ otelHeapWaitTime.record(waitTimeNanos / 1_000_000_000.0d);
+ }
+ }
+
+ @Override
+ protected void recordDirectWaitTime(long waitTimeNanos) {
+ if (waitTimeNanos == Long.MAX_VALUE) {
+ directTimeoutTotal.inc();
+ otelDirectTimeoutTotal.add(1);
+ } else {
+ directWaitTimeMs.observe(TimeUnit.NANOSECONDS.toMillis(waitTimeNanos));
+ otelDirectWaitTime.record(waitTimeNanos / 1_000_000_000.0d);
+ }
+ }
+
+ @Override
+ public void close() {
+ super.close();
+ unregister(heapMemoryUsedBytes);
+ unregister(heapMemoryLimitBytes);
+ unregister(directMemoryUsedBytes);
+ unregister(directMemoryLimitBytes);
+ unregister(heapQueueSize);
+ unregister(heapQueueMaxSize);
+ unregister(directQueueSize);
+ unregister(directQueueMaxSize);
+ unregister(heapWaitTimeMs);
+ unregister(directWaitTimeMs);
+ unregister(heapTimeoutTotal);
+ unregister(directTimeoutTotal);
+ otelHeapMemoryUsedGauge.close();
+ otelDirectMemoryUsedGauge.close();
+ otelHeapQueueSize.close();
+ otelDirectQueueSize.close();
+ }
+
+ /**
+ * Estimate the heap memory size of a topic list.
+ * @param topicList the topic list to estimate
+ * @return the estimated heap memory size in bytes
+ */
+ public static long estimateTopicListSize(List<String> topicList) {
+ return topicList.stream()
+ .mapToLong(ByteBufUtil::utf8Bytes) // convert character count to bytes
+ // add 32 bytes overhead for each entry
+ // 16 bytes for object header, 16 bytes for java.lang.String fields
+ .map(n -> n + 32)
+ .sum();
+ }
Review Comment:
It says "convert character count to bytes" which should be clear.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2479078939
##########
pulsar-common/src/main/java/org/apache/pulsar/common/semaphore/AsyncSemaphoreImpl.java:
##########
@@ -0,0 +1,330 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.common.semaphore;
+
+import io.netty.util.concurrent.DefaultThreadFactory;
+import java.util.Queue;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicLongFieldUpdater;
+import java.util.function.BooleanSupplier;
+import java.util.function.LongConsumer;
+import org.apache.pulsar.common.util.Runnables;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of AsyncSemaphore with timeout and queue size limits.
+ */
+public class AsyncSemaphoreImpl implements AsyncSemaphore, AutoCloseable {
+ private static final Logger log = LoggerFactory.getLogger(AsyncSemaphoreImpl.class);
+
+ private final AtomicLong availablePermits;
+ private final Queue<PendingRequest> queue;
+ private final long maxPermits;
+ private final long timeoutMillis;
+ private final ScheduledExecutorService executor;
+ private final boolean shutdownExecutor;
+ private final LongConsumer queueLatencyRecorder;
+ private final AtomicBoolean closed = new AtomicBoolean(false);
+ private final Runnable processQueueRunnable = Runnables.catchingAndLoggingThrowables(this::internalProcessQueue);
+
+ /**
+ * Creates an AsyncSemaphoreImpl with the given parameters.
+ * @param maxPermits max number of permits available for acquisition, set to <= 0 for unbounded semaphore (not
+ * recommended)
+ * @param maxQueueSize max number of requests that can be queued, set to <= 0 for unbounded queue (not recommended)
+ * @param timeoutMillis timeout in milliseconds for acquiring permits
+ */
+ public AsyncSemaphoreImpl(long maxPermits, int maxQueueSize, long timeoutMillis) {
+ this(maxPermits, maxQueueSize, timeoutMillis, maxPermits > 0 ? createExecutor() : null, maxPermits > 0, null);
+ }
+
+ /**
+ * Creates an AsyncSemaphoreImpl with the given parameters.
+ * @param maxPermits max number of permits available for acquisition, set to <= 0 for unbounded semaphore (not
+ * recommended)
+ * @param maxQueueSize max number of requests that can be queued, set to <= 0 for unbounded queue (not recommended)
+ * @param timeoutMillis timeout in milliseconds for acquiring permits
+ * @param executor executor service to use for scheduling timeouts, it is expected to be single threaded
+ * @param queueLatencyRecorder consumer to record queue latency, Long.MAX_VALUE is used for requests that timed out
+ */
+ public AsyncSemaphoreImpl(long maxPermits, int maxQueueSize, long timeoutMillis,
+ ScheduledExecutorService executor, LongConsumer queueLatencyRecorder) {
+ this(maxPermits, maxQueueSize, timeoutMillis, executor, false, queueLatencyRecorder);
+ }
+
+ AsyncSemaphoreImpl(long maxPermits, int maxQueueSize, long timeoutMillis, ScheduledExecutorService executor,
+ boolean shutdownExecutor, LongConsumer queueLatencyRecorder) {
+ this.availablePermits = new AtomicLong(maxPermits);
+ this.maxPermits = maxPermits;
+ this.queue = maxQueueSize > 0 ? new ArrayBlockingQueue<>(maxQueueSize) : new LinkedBlockingQueue<>();
+ this.timeoutMillis = timeoutMillis;
+ this.executor = executor;
+ this.shutdownExecutor = shutdownExecutor;
+ this.queueLatencyRecorder = queueLatencyRecorder;
+ }
+
+ private static ScheduledExecutorService createExecutor() {
+ return Executors.newSingleThreadScheduledExecutor(
+ new DefaultThreadFactory("async-semaphore-executor"));
+ }
+
+ @Override
+ public CompletableFuture<AsyncSemaphorePermit> acquire(long permits, BooleanSupplier isCancelled) {
+ return internalAcquire(permits, permits, isCancelled);
+ }
+
+ private CompletableFuture<AsyncSemaphorePermit> internalAcquire(long permits, long acquirePermits,
+ BooleanSupplier isCancelled) {
+ validatePermits(permits);
+
+ // if maximum permits is <= 0, then the semaphore is unbounded
+ if (isUnbounded()) {
+ return CompletableFuture.completedFuture(new SemaphorePermit(permits));
+ }
+
+ CompletableFuture<AsyncSemaphorePermit> future = new CompletableFuture<>();
+
+ if (closed.get()) {
+ future.completeExceptionally(new PermitAcquireAlreadyClosedException("Semaphore is closed"));
+ return future;
+ }
+
+ PendingRequest request = new PendingRequest(permits, acquirePermits, future, isCancelled);
+ if (!queue.offer(request)) {
+ future.completeExceptionally(new PermitAcquireQueueFullException(
+ "Semaphore queue is full"));
+ return future;
+ }
+ // Schedule timeout
+ ScheduledFuture<?> timeoutTask = executor.schedule(() -> {
+ if (!request.future.isDone() && queue.remove(request)) {
+ // timeout is recorded with Long.MAX_VALUE as the age
+ recordQueueLatency(Long.MAX_VALUE);
+ // also record the time in the queue
+ recordQueueLatency(request.getAgeNanos());
+ future.completeExceptionally(new PermitAcquireTimeoutException(
+ "Permit acquisition timed out"));
+ // the next request might have smaller permits and that might be processed
+ processQueue();
+ }
+ }, timeoutMillis, TimeUnit.MILLISECONDS);
+ request.setTimeoutTask(timeoutTask);
+
+ processQueue();
+ return future;
+ }
+
+ private void validatePermits(long permits) {
+ if (permits < 0) {
+ throw new IllegalArgumentException("Invalid negative permits value: " + permits);
+ }
+ if (!isUnbounded() && permits > maxPermits) {
+ throw new IllegalArgumentException(
+ "Requested permits=" + permits + " is larger than maxPermits=" + maxPermits);
+ }
+ }
+
+ private boolean isUnbounded() {
+ return maxPermits <= 0;
+ }
+
+ private void recordQueueLatency(long ageNanos) {
+ if (queueLatencyRecorder != null) {
+ queueLatencyRecorder.accept(ageNanos);
+ }
+ }
+
+ @Override
+ public CompletableFuture<AsyncSemaphorePermit> update(AsyncSemaphorePermit permit, long newPermits,
+ BooleanSupplier isCancelled) {
+ validatePermits(newPermits);
+ if (isUnbounded()) {
+ return CompletableFuture.completedFuture(new SemaphorePermit(newPermits));
+ }
+ long oldPermits = permit.getPermits();
+ long additionalPermits = newPermits - oldPermits;
+ // mark the old permits as released without adding the permits to availablePermits
+ castToImplementation(permit).releasePermits();
+ if (additionalPermits > 0) {
+ return internalAcquire(newPermits, additionalPermits, isCancelled);
+ } else {
+ // new permits are less than the old ones, so we return the difference
+ availablePermits.addAndGet(-additionalPermits);
+ processQueue();
+ // return the new permits immediately
+ return CompletableFuture.completedFuture(new SemaphorePermit(newPermits));
Review Comment:
When permits are transferred to another instance, the permits are released from the instance. Calling release again won't release the permits again.
There was a bug where permits where released before the updated permits were successfully acquired. That is fixed and there's new test cases to cover it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace/CommandWatchTopicList [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #24833:
URL: https://github.com/apache/pulsar/pull/24833#discussion_r2491322763
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/TopicListService.java:
##########
@@ -101,6 +112,7 @@ public void accept(String topicName, NotificationType notificationType) {
private final boolean enableSubscriptionPatternEvaluation;
private final int maxSubscriptionPatternLength;
private final ConcurrentLongHashMap<CompletableFuture<TopicListWatcher>> watchers;
+ private final Backoff retryBackoff;
public TopicListService(PulsarService pulsar, ServerCnx connection,
Review Comment:
> @lhotari The command should not be limited, because `MultiTopicsConsumer` and `MultiTopicsProducer` need to receive this command once the topics' partition is changed. The regexp consumer is also relies on the command.
@poorbarcode There's perhaps some misunderstanding. Memory limiting won't prevent this command from being executed unless it times out or the queue fills up. It is absolutely necessary to handle also CommandWatchTopicListSuccess / CommandWatchTopicUpdate related commands so that the broker doesn't run out of memory and that's also explained in [PIP-442](https://github.com/apache/pulsar/blob/master/pip/pip-442.md). Handling CommandWatchTopicListSuccess / CommandWatchTopicUpdate related commands with the memory limiter will help the broker remain stable regardless of a large amount of regex consumers connecting at once. When the limit is reached, it will queue up the requests until other requests have completed.
Does this make sense?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org