You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "sindhushreem (via GitHub)" <gi...@apache.org> on 2023/01/27 04:31:08 UTC

[GitHub] [pulsar] sindhushreem opened a new issue, #19334: bookkeeper is running out of direct memory

sindhushreem opened a new issue, #19334:
URL: https://github.com/apache/pulsar/issues/19334

   ### Search before asking
   
   - [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
   
   
   ### Version
   
   2.8.2
   
   ### Minimal reproduce step
   
   bookeeper runnning for some days and ran out of direct memory .
   
   ### What did you expect to see?
   
   Bookkeeper should be running without any error .
   
   ### What did you see instead?
   
   INFO  org.apache.bookkeeper.bookie.JournalChannel - Opening journal data/bookkeeper/journal/current/181400c6508.txn
   2022-11-14T22:01:22,123 [main] ERROR org.apache.bookkeeper.common.component.AbstractLifecycleComponent - Failed to start Component: bookie-server
   java.lang.OutOfMemoryError: Direct buffer memory
   	at java.nio.Bits.reserveMemory(Bits.java:695) ~[?:1.8.0_332]
   	at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) ~[?:1.8.0_332]
   	at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) ~[?:1.8.0_332]
   	at [sun.nio.ch](http://sun.nio.ch/).Util.getTemporaryDirectBuffer(Util.java:247) ~[?:1.8.0_332]
   	at [sun.nio.ch](http://sun.nio.ch/).IOUtil.read(IOUtil.java:195) ~[?:1.8.0_332]
   	at [sun.nio.ch](http://sun.nio.ch/).FileChannelImpl.read(FileChannelImpl.java:159) ~[?:1.8.0_332]
   	at org.apache.bookkeeper.bookie.JournalChannel.read(JournalChannel.java:261) ~[org.apache.bookkeeper-bookkeeper-server-4.14.3.jar:4.14.3]
   	at org.apache.bookkeeper.bookie.Journal.fullRead(Journal.java:1194) ~[org.apache.bookkeeper-bookkeeper-server-4.14.3.jar:4.14.3]
   	at org.apache.bookkeeper.bookie.Journal.scanJournal(Journal.java:833) ~[org.apache.bookkeeper-bookkeeper-server-4.14.3.jar:4.14.3]
   	at org.apache.bookkeeper.bookie.Bookie.replay(Bookie.java:995) ~[org.apache.bookkeeper-bookkeeper-server-4.14.3.jar:4.14.3]
   	at org.apache.bookkeeper.bookie.Bookie.readJournal(Bookie.java:961) ~[org.apache.bookkeeper-bookkeeper-server-4.14.3.jar:4.14.3]
   	at org.apache.bookkeeper.bookie.Bookie.start(Bookie.java:1015) ~[org.apache.bookkeeper-bookkeeper-server-4.14.3.jar:4.14.3]
   	at org.apache.bookkeeper.proto.BookieServer.start(BookieServer.java:156) ~[org.apache.bookkeeper-bookkeeper-server-4.14.3.jar:4.14.3]
   	at org.apache.bookkeeper.server.service.BookieService.doStart(BookieService.java:68) ~[org.apache.bookkeeper-bookkeeper-server-4.14.3.jar:4.14.3]
   	at org.apache.bookkeeper.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:83) ~[org.apache.bookkeeper-bookkeeper-common-4.14.3.jar:4.14.3]
   	at org.apache.bookkeeper.common.component.LifecycleComponentStack.lambda$start$4(LifecycleComponentStack.java:144) ~[org.apache.bookkeeper-bookkeeper-common-4.14.3.jar:4.14.3]
   	at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:406) [com.google.guava-guava-30.1-jre.jar:?]
   	at org.apache.bookkeeper.common.component.LifecycleComponentStack.start(LifecycleComponentStack.java:144) [org.apache.bookkeeper-bookkeeper-common-4.14.3.jar:4.14.3]
   	at org.apache.bookkeeper.common.component.ComponentStarter.startComponent(ComponentStarter.java:85) [org.apache.bookkeeper-bookkeeper-common-4.14.3.jar:4.14.3]
   	at org.apache.bookkeeper.server.Main.doMain(Main.java:234) [org.apache.bookkeeper-bookkeeper-server-4.14.3.jar:4.14.3]
   	at org.apache.bookkeeper.server.Main.main(Main.java:208) [org.apache.bookkeeper-bookkeeper-server-4.14.3.jar:4.14.3]
   2022-11-14T22:01:22,129 [main] ERROR org.apache.bookkeeper.common.component.AbstractLifecycleComponent - Calling uncaughtExceptionHandler
   2022-11-14T22:01:22,129 [main] ERROR org.apache.bookkeeper.common.component.ComponentStarter - Triggered exceptionHandler of Component: bookie-server because of Exception in Thread: Thread[main,5,main]
   java.lang.OutOfMemoryError: Direct buffer memory
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [ ] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] sindhushreem commented on issue #19334: bookkeeper is running out of direct memory

Posted by "sindhushreem (via GitHub)" <gi...@apache.org>.
sindhushreem commented on issue #19334:
URL: https://github.com/apache/pulsar/issues/19334#issuecomment-1447573245

   > Hi @sindhushreem, thank you for raising this issue. Could you share your direct memory configuration and the heap dump if possible?
   
   Hi Hang ,
   I have shared the direct memory configuration and heap dump with @codelipenghui 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bookkeeper is running out of direct memory [pulsar]

Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat closed issue #19334: bookkeeper is running out of direct memory
URL: https://github.com/apache/pulsar/issues/19334


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] github-actions[bot] commented on issue #19334: bookkeeper is running out of direct memory

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #19334:
URL: https://github.com/apache/pulsar/issues/19334#issuecomment-1447453947

   The issue had no activity for 30 days, mark with Stale label.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bookkeeper is running out of direct memory [pulsar]

Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat commented on issue #19334:
URL: https://github.com/apache/pulsar/issues/19334#issuecomment-6386663035

   Thanks @sindhushreem for the report. This was reported on Pulsar 2.8, which is no longer supported; the supported releases are 4.0.x, 4.2.x and 5.0.x. Much of the code involved has changed since, and the problem hasn't been confirmed on a current release, so I'm closing it as part of cleaning up old reports. If you still see it on a supported release, please open a new issue with the version and steps to reproduce, and link this one.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] github-actions[bot] commented on issue #19334: bookkeeper is running out of direct memory

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #19334:
URL: https://github.com/apache/pulsar/issues/19334#issuecomment-1492786715

   The issue had no activity for 30 days, mark with Stale label.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] hangc0276 commented on issue #19334: bookkeeper is running out of direct memory

Posted by "hangc0276 (via GitHub)" <gi...@apache.org>.
hangc0276 commented on issue #19334:
URL: https://github.com/apache/pulsar/issues/19334#issuecomment-1407319897

   Hi @sindhushreem, thank you for raising this issue. Could you share your direct memory configuration and the heap dump if possible?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org