You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@pulsar.apache.org by "mattisonchao (via GitHub)" <gi...@apache.org> on 2023/07/17 10:53:50 UTC

[GitHub] [pulsar] mattisonchao opened a new pull request, #20823: [fix][broker] Fix direct memory leak by delayed index OutOfDirectMemory

mattisonchao opened a new pull request, #20823:
URL: https://github.com/apache/pulsar/pull/20823

   ### Motivation
   
   [There](https://github.com/apache/pulsar/blob/57fbee446aebb5c9f4db3bed559cb7bef3749731/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractBaseDispatcher.java#L205C16-L205C16) is a memory leak if the delayed tracker throws an `OutOfDirectMemory` exception. The entries will never release.
   
   ```
   
   Jul 16, 2023 @ 18:35:58.575 | 2023-07-16T10:35:58,574+0000 [broker-topic-workers-OrderedExecutor-0-0] ERROR org.apache.bookkeeper.common.util.SingleThreadExecutor - Error while running task: failed to allocate 16777216 byte(s) of direct memory (used: 847249415, max: 858993459)
     | Jul 16, 2023 @ 18:35:58.575 | at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.93.Final.jar:4.1.93.Final]
     | Jul 16, 2023 @ 18:35:58.575 | at org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers.trySendMessagesToConsumers(PersistentDispatcherMultipleConsumers.java:717) ~[io.streamnative-pulsar-broker-3.0.0.4.jar:3.0.0.4]
     | Jul 16, 2023 @ 18:35:58.575 | at org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers.trackDelayedDelivery(PersistentDispatcherMultipleConsumers.java:1068) ~[io.streamnative-pulsar-broker-3.0.0.4.jar:3.0.0.4]
     | Jul 16, 2023 @ 18:35:58.575 | at io.netty.buffer.PoolArena.allocate(PoolArena.java:144) ~[io.netty-netty-buffer-4.1.93.Final.jar:4.1.93.Final]
     | Jul 16, 2023 @ 18:35:58.575 | at io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:774) ~[io.netty-netty-common-4.1.93.Final.jar:4.1.93.Final]
     | Jul 16, 2023 @ 18:35:58.575 | at org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers.sendMessagesToConsumers(PersistentDispatcherMultipleConsumers.java:622) ~[io.streamnative-pulsar-broker-3.0.0.4.jar:3.0.0.4]
     | Jul 16, 2023 @ 18:35:58.575 | at org.apache.pulsar.broker.delayed.InMemoryDelayedDeliveryTracker.addMessage(InMemoryDelayedDeliveryTracker.java:81) ~[io.streamnative-pulsar-broker-3.0.0.4.jar:3.0.0.4]
     | Jul 16, 2023 @ 18:35:58.575 | at io.netty.buffer.PoolArena.allocate(PoolArena.java:129) ~[io.netty-netty-buffer-4.1.93.Final.jar:4.1.93.Final]
     | Jul 16, 2023 @ 18:35:58.575 | at io.netty.buffer.PoolArena$DirectArena.newUnpooledChunk(PoolArena.java:690) ~[io.netty-netty-buffer-4.1.93.Final.jar:4.1.93.Final]
     | Jul 16, 2023 @ 18:35:58.575 | io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 847249415, max: 858993459)
     | Jul 16, 2023 @ 18:35:58.575 | at org.apache.bookkeeper.common.util.SingleThreadExecutor.safeRunTask(SingleThreadExecutor.java:137) ~[org.apache.bookkeeper-bookkeeper-common-4.16.2.jar:4.16.2] 
     | Jul 16, 2023 @ 18:35:58.575 | at org.apache.pulsar.common.util.collections.TripleLongPriorityQueue.add(TripleLongPriorityQueue.java:94) ~[io.streamnative-pulsar-common-3.0.0.4.jar:3.0.0.4]
     | Jul 16, 2023 @ 18:35:58.575 | at io.netty.buffer.PoolArena.allocateHuge(PoolArena.java:226) ~[io.netty-netty-buffer-4.1.93.Final.jar:4.1.93.Final]
     | Jul 16, 2023 @ 18:35:58.575 | at org.apache.bookkeeper.common.util.SingleThreadExecutor.run(SingleThreadExecutor.java:113) ~[org.apache.bookkeeper-bookkeeper-common-4.16.2.jar:4.16.2]
   ```
   
   
   ### Modifications
   
   - Add exception catcher to catch Direct memory OOM exception and then try to recover it since we control the direct memory ourselves.
   
   
   ### Verifying this change
   
   - [x] Make sure that the change passes the CI checks.
   
   ### Documentation
   
   <!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
   
   - [ ] `doc` <!-- Your PR contains doc changes. -->
   - [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
   - [x] `doc-not-needed` <!-- Your PR changes do not impact docs -->
   - [ ] `doc-complete` <!-- Docs have been already added -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] poorbarcode commented on a diff in pull request #20823: [fix][broker] Fix direct memory leak by delayed index OutOfDirectMemory

Posted by "poorbarcode (via GitHub)" <gi...@apache.org>.

poorbarcode commented on code in PR #20823:
URL: https://github.com/apache/pulsar/pull/20823#discussion_r1266408122


##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentDispatcherMultipleConsumers.java:
##########
@@ -1065,7 +1066,17 @@ public boolean trackDelayedDelivery(long ledgerId, long entryId, MessageMetadata
             delayedDeliveryTracker.get().resetTickTime(topic.getDelayedDeliveryTickTimeMillis());
 
             long deliverAtTime = msgMetadata.hasDeliverAtTime() ? msgMetadata.getDeliverAtTime() : -1L;
-            return delayedDeliveryTracker.get().addMessage(ledgerId, entryId, deliverAtTime);
+            try {
+                return delayedDeliveryTracker.get().addMessage(ledgerId, entryId, deliverAtTime);
+            } catch (OutOfDirectMemoryError ex) {
+                log.error("Out of direct memory while trying add message index to delayed tracker."
+                        + " fallback to redelivery messages(heap) and waiting for enough direct memory space."
+                        + " error info: {}", ex.getMessage());
+                // fall back to redelivery queue(heap) waiting for enough direct memory.
+                redeliveryMessages.add(ledgerId, entryId);

Review Comment:
   This makes the delayed messages never delay, right?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org