You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2020/06/14 07:55:58 UTC
[GitHub] [pulsar] xuesong172500 opened a new issue #7270: bookie oom, auto exited
xuesong172500 opened a new issue #7270:
URL: https://github.com/apache/pulsar/issues/7270
**Describe the bug**
15:15:56.967 [db-storage-9-1] INFO org.apache.bookkeeper.bookie.EntryLogManagerForSingleEntryLog - Synced entry logger 0 to disk.
15:16:23.548 [BookieWriteThreadPool-OrderedExecutor-0-0] INFO org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage - Write cache is full, triggering flush
15:16:53.837 [BookieWriteThreadPool-OrderedExecutor-0-0] INFO org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage - Write cache is full, triggering flush
15:18:42.667 [BookieReadThreadPool-OrderedExecutor-0-0] ERROR org.apache.bookkeeper.proto.BookieServer - Unable to allocate memory, exiting bookie
io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 2130706463, max: 2147483648)
at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:726) ~[io.netty-netty-common-4.1.45.Final.jar:4.1.45.Final]
at io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:681) ~[io.netty-netty-common-4.1.45.Final.jar:4.1.45.Final]
at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:758) ~[io.netty-netty-buffer-4.1.45.Final.jar:4.1.45.Final]
at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:734) ~[io.netty-netty-buffer-4.1.45.Final.jar:4.1.45.Final]
at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:245) ~[io.netty-netty-buffer-4.1.45.Final.jar:4.1.45.Final]
at io.netty.buffer.PoolArena.allocate(PoolArena.java:227) ~[io.netty-netty-buffer-4.1.45.Final.jar:4.1.45.Final]
at io.netty.buffer.PoolArena.allocate(PoolArena.java:147) ~[io.netty-netty-buffer-4.1.45.Final.jar:4.1.45.Final]
at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:342) ~[io.netty-netty-buffer-4.1.45.Final.jar:4.1.45.Final]
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187) ~[io.netty-netty-buffer-4.1.45.Final.jar:4.1.45.Final]
at org.apache.bookkeeper.common.allocator.impl.ByteBufAllocatorImpl.newDirectBuffer(ByteBufAllocatorImpl.java:164) [org.apache.bookkeeper-bookkeeper-common-allocator-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.common.allocator.impl.ByteBufAllocatorImpl.newDirectBuffer(ByteBufAllocatorImpl.java:158) [org.apache.bookkeeper-bookkeeper-common-allocator-4.10.0.jar:4.10.0]
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187) [io.netty-netty-buffer-4.1.45.Final.jar:4.1.45.Final]
at org.apache.bookkeeper.bookie.storage.ldb.ReadCache.get(ReadCache.java:145) [org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.bookie.storage.ldb.SingleDirectoryDbLedgerStorage.getEntry(SingleDirectoryDbLedgerStorage.java:429) [org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage.getEntry(DbLedgerStorage.java:199) [org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.bookie.LedgerDescriptorImpl.readEntry(LedgerDescriptorImpl.java:159) [org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.bookie.Bookie.readEntry(Bookie.java:1420) [org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.proto.ReadEntryProcessor.processPacket(ReadEntryProcessor.java:80) [org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.proto.PacketProcessorBase.safeRun(PacketProcessorBase.java:85) [org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) [org.apache.bookkeeper-bookkeeper-common-4.10.0.jar:4.10.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_101]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_101]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.45.Final.jar:4.1.45.Final]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_101]
**To Reproduce**
Steps to reproduce the behavior:
1. bin/pulsar-perf produce -threads 2 -r 50000 persistent://public/default/test-string100
2. bin/pulsar-perf consume -q 100 -n 100 -st Shared -s consumerSych persistent://public/default/test-string100
3. Topic has 100 partitions
**Expected behavior**
A clear and concise description of what you expected to happen.
**Screenshots**
If applicable, add screenshots to help explain your problem.
**Desktop (please complete the following information):**
- OS: [e.g. iOS]
**Additional context**
Add any other context about the problem here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] jiazhai commented on issue #7270: bookie oom, auto exited
Posted by GitBox <gi...@apache.org>.
jiazhai commented on issue #7270:
URL: https://github.com/apache/pulsar/issues/7270#issuecomment-649874967
```
15:16:23.548 [BookieWriteThreadPool-OrderedExecutor-0-0] INFO org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage - Write cache is full, triggering flush
```
Regarding this log, we could try to set these vars.
```
# Size of Write Cache. Memory is allocated from JVM direct memory.
# Write cache is used to buffer entries before flushing into the entry log
# For good performance, it should be big enough to hold a substantial amount
# of entries in the flush interval
# By default it will be allocated to 1/4th of the available direct memory
dbStorage_writeCacheMaxSizeMb=
# Size of Read cache. Memory is allocated from JVM direct memory.
# This read cache is pre-filled doing read-ahead whenever a cache miss happens
# By default it will be allocated to 1/4th of the available direct memory
dbStorage_readAheadCacheMaxSizeMb=
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] cherrizj commented on issue #7270: bookie oom, auto exited
Posted by GitBox <gi...@apache.org>.
cherrizj commented on issue #7270:
URL: https://github.com/apache/pulsar/issues/7270#issuecomment-663525638
05:42:55.378 [bookie-io-1-27] ERROR org.apache.bookkeeper.proto.BookieServer - Unable to allocate memory, exiting bookie
io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 2147483648, max: 2147483648)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org