You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@bookkeeper.apache.org by GitBox <gi...@apache.org> on 2020/02/21 07:53:25 UTC
[GitHub] [bookkeeper] aloyszhang opened a new issue #2269: Bookie shutdown
due to OutOfDirectMemoryError
aloyszhang opened a new issue #2269: Bookie shutdown due to OutOfDirectMemoryError
URL: https://github.com/apache/bookkeeper/issues/2269
**BUG REPORT**
Bookie shutdown due to OutOfDirectMemoryError
***Describe the bug***
Bookie shutdown when pulsar continuous send message.
```
ERROR [bookie-io-1-19:BookieRequestHandler@70] - Unhandled exception occurred in I/O thread or handler
io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 21474836480, max: 21474836480)
at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:655)
at io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:610)
at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:769)
at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:745)
at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:244)
at io.netty.buffer.PoolArena.allocate(PoolArena.java:226)
at io.netty.buffer.PoolArena.reallocate(PoolArena.java:397)
at io.netty.buffer.PooledByteBuf.capacity(PooledByteBuf.java:118)
at io.netty.buffer.AbstractByteBuf.ensureWritable0(AbstractByteBuf.java:299)
at io.netty.buffer.AbstractByteBuf.ensureWritable(AbstractByteBuf.java:278)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1103)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1096)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1087)
at io.netty.handler.codec.ByteToMessageDecoder$1.cumulate(ByteToMessageDecoder.java:93)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:433)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:330)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
```
***To Reproduce***
In my case, there are 4 bookies start with the following configuration.
bkenv.sh
```
BOOKIE_MAX_HEAP_MEMORY=10g
BOOKIE_MIN_HEAP_MEMORY=10g
BOOKIE_MAX_DIRECT_MEMORY=20g
```
bk_server.sh
```
journalDirectories=/data2/bookkeeper/journal,/data3/bookkeeper/journal,/data4/bookkeeper/journal
ledgerDirectories=/data1/bk-data,/data5/bk-data,/data7/bk-data,/data8/bk-data,/data9/bk-data
```
Directory data1,data2,data3,data4,data5,data6,data7,date8,data9 are on different disk with individual disk driver.
Then start client (actuallly pulsar) to send message by pulsar-perf tool with the command :
`bin/pulsar-perf produce -threads 6 -u pulsar://10.56.204.197:6650,10.56.204.198:6650,10.56.205.196:6650 -o 10000 -n 1 -s 1000 -r 3000000 persistent://tenant-bk/namespace-bk/topic-bk`
***Expected behavior***
BookKeeper should have mechanism of rate limit to avoid OutOfDirectMemoryError happends.
***Screenshots***

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [bookkeeper] aloyszhang commented on issue #2269: Bookie shutdown
due to OutOfDirectMemoryError
Posted by GitBox <gi...@apache.org>.
aloyszhang commented on issue #2269: Bookie shutdown due to OutOfDirectMemoryError
URL: https://github.com/apache/bookkeeper/issues/2269#issuecomment-593913483
Thanks for your suggestion @RaulGracia
Since I wanna to run Pulsar on BookKeeper.
I turn configuration to Pulsar suggested values which take `DbLedgerStorage` as Ledger Storage as @eolivelli suggested.
It runs well for the passed few days.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [bookkeeper] aloyszhang edited a comment on issue #2269: Bookie
shutdown due to OutOfDirectMemoryError
Posted by GitBox <gi...@apache.org>.
aloyszhang edited a comment on issue #2269: Bookie shutdown due to OutOfDirectMemoryError
URL: https://github.com/apache/bookkeeper/issues/2269#issuecomment-590142877
Thanks for your reply.
Yes , I do write huge message abount 20,0000 qps with each message has 1KB payload。
I use the configuration with default values on "master" branch.
I will try again on configuration shipped with Pulsar.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [bookkeeper] aloyszhang edited a comment on issue #2269: Bookie
shutdown due to OutOfDirectMemoryError
Posted by GitBox <gi...@apache.org>.
aloyszhang edited a comment on issue #2269: Bookie shutdown due to OutOfDirectMemoryError
URL: https://github.com/apache/bookkeeper/issues/2269#issuecomment-590142877
Thanks for your reply.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [bookkeeper] aloyszhang commented on issue #2269: Bookie shutdown
due to OutOfDirectMemoryError
Posted by GitBox <gi...@apache.org>.
aloyszhang commented on issue #2269: Bookie shutdown due to OutOfDirectMemoryError
URL: https://github.com/apache/bookkeeper/issues/2269#issuecomment-590142877
Thanks , I'll try.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [bookkeeper] aloyszhang edited a comment on issue #2269: Bookie
shutdown due to OutOfDirectMemoryError
Posted by GitBox <gi...@apache.org>.
aloyszhang edited a comment on issue #2269: Bookie shutdown due to OutOfDirectMemoryError
URL: https://github.com/apache/bookkeeper/issues/2269#issuecomment-590142877
Thanks for your reply.
Yes , I do write huge message abount 20,0000 qps with each message has 1KB payload。
I use the configuration with default values on "master" branch.
I will try again with configuration shipped with Pulsar.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [bookkeeper] aloyszhang edited a comment on issue #2269: Bookie
shutdown due to OutOfDirectMemoryError
Posted by GitBox <gi...@apache.org>.
aloyszhang edited a comment on issue #2269: Bookie shutdown due to OutOfDirectMemoryError
URL: https://github.com/apache/bookkeeper/issues/2269#issuecomment-589910246
This problem happens when run pulsar on BookKeeper with ensemble 3, write quorum 3 and ack quorum 2.
Data are send to 3 bookies at the same time, so ,I think the same problem will happeds with a single bookie.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [bookkeeper] RaulGracia commented on issue #2269: Bookie shutdown
due to OutOfDirectMemoryError
Posted by GitBox <gi...@apache.org>.
RaulGracia commented on issue #2269: Bookie shutdown due to OutOfDirectMemoryError
URL: https://github.com/apache/bookkeeper/issues/2269#issuecomment-593888409
@aloyszhang @eolivelli I have seen this problem as well. In our case, we have prevented this from happening by using `InterleavedLedgerStorage` instead of the default one (`SortedLedgerStorage`), as it does not contain `EntryMemTable` objects that seem to use significant amount of memory (tested with Bookkeeper 4.9.2). Of course, this decision comes at the cost of read performance, but in our case we can afford it, as we only read from Bookkeeper for recovery after a crash. Since we shifted the ledger type, we have not seen more memory issues like this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [bookkeeper] aloyszhang commented on issue #2269: Bookie shutdown
due to OutOfDirectMemoryError
Posted by GitBox <gi...@apache.org>.
aloyszhang commented on issue #2269: Bookie shutdown due to OutOfDirectMemoryError
URL: https://github.com/apache/bookkeeper/issues/2269#issuecomment-589641784
bookkeeper-server-4.11.0-SNAPSHOT @eolivelli
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [bookkeeper] eolivelli commented on issue #2269: Bookie shutdown
due to OutOfDirectMemoryError
Posted by GitBox <gi...@apache.org>.
eolivelli commented on issue #2269: Bookie shutdown due to OutOfDirectMemoryError
URL: https://github.com/apache/bookkeeper/issues/2269#issuecomment-589679292
Are you able to reproduce the problem with a single bookie ?
I mean, in order to create a simpler reproducer ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [bookkeeper] aloyszhang edited a comment on issue #2269: Bookie
shutdown due to OutOfDirectMemoryError
Posted by GitBox <gi...@apache.org>.
aloyszhang edited a comment on issue #2269: Bookie shutdown due to OutOfDirectMemoryError
URL: https://github.com/apache/bookkeeper/issues/2269#issuecomment-590142877
Thanks for your reply.
Yes , I do write huge message abount 20,0000 qps with each message has 1KB payload。
I use the configuration with default values on "master" branch.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [bookkeeper] eolivelli commented on issue #2269: Bookie shutdown
due to OutOfDirectMemoryError
Posted by GitBox <gi...@apache.org>.
eolivelli commented on issue #2269: Bookie shutdown due to OutOfDirectMemoryError
URL: https://github.com/apache/bookkeeper/issues/2269#issuecomment-590053233
We have a few features to prevent this situation.
Are you writing huge messages ?
Are you running the Bookie with the configuration shipped with Pulsar or the default values on "master" branch ?
You can try to tune the Bookie using Pulsar suggested values.
You can check all tunables about the maximum number of concurrent writes (pendingAdds) in this file:
https://github.com/apache/bookkeeper/blob/master/bookkeeper-server/src/main/java/org/apache/bookkeeper/conf/ServerConfiguration.java#L198
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [bookkeeper] eolivelli commented on issue #2269: Bookie shutdown
due to OutOfDirectMemoryError
Posted by GitBox <gi...@apache.org>.
eolivelli commented on issue #2269: Bookie shutdown due to OutOfDirectMemoryError
URL: https://github.com/apache/bookkeeper/issues/2269#issuecomment-589639662
Which version of BK do you have?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [bookkeeper] aloyszhang commented on issue #2269: Bookie shutdown
due to OutOfDirectMemoryError
Posted by GitBox <gi...@apache.org>.
aloyszhang commented on issue #2269: Bookie shutdown due to OutOfDirectMemoryError
URL: https://github.com/apache/bookkeeper/issues/2269#issuecomment-589910246
This problem happens when run pulsar on BookKeeper with ensemble 3, write quorum 3 and ack quorum 2. Data are send to 3 bookies at the same time, so ,I think the same problem will happeds with a single bookie.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services