You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/02/25 07:13:16 UTC
[GitHub] [pulsar] armandxu opened a new issue #14468: bookie-io-1-9] ERROR org.apache.bookkeeper.proto.BookieServer - Unable to allocate memory, exiting bookie
armandxu opened a new issue #14468:
URL: https://github.com/apache/pulsar/issues/14468
**Describe the bug**
when storage is full，then bookie has an error：
2022-02-24T23:13:39,590+0800 [bookie-io-1-9] ERROR org.apache.bookkeeper.proto.BookieServer - Unable to allocate memory, exiting bookie
io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 23605542941, max: 23622320128)
at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:802) ~[io.netty-netty-common-4.1.72.Final.jar:4.1.72.Final]
at io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:731) ~[io.netty-netty-common-4.1.72.Final.jar:4.1.72.Final]
at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:648) ~[io.netty-netty-buffer-4.1.72.Final.jar:4.1.72.Final]
at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:623) ~[io.netty-netty-buffer-4.1.72.Final.jar:4.1.72.Final]
at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:202) ~[io.netty-netty-buffer-4.1.72.Final.jar:4.1.72.Final]
at io.netty.buffer.PoolArena.tcacheAllocateNormal(PoolArena.java:186) ~[io.netty-netty-buffer-4.1.72.Final.jar:4.1.72.Final]
at io.netty.buffer.PoolArena.allocate(PoolArena.java:136) ~[io.netty-netty-buffer-4.1.72.Final.jar:4.1.72.Final]
at io.netty.buffer.PoolArena.allocate(PoolArena.java:126) ~[io.netty-netty-buffer-4.1.72.Final.jar:4.1.72.Final]
at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:395) ~[io.netty-netty-buffer-4.1.72.Final.jar:4.1.72.Final]
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188) ~[io.netty-netty-buffer-4.1.72.Final.jar:4.1.72.Final]
at org.apache.bookkeeper.common.allocator.impl.ByteBufAllocatorImpl.newDirectBuffer(ByteBufAllocatorImpl.java:163) [org.apache.bookkeeper-bookkeeper-common-allocator-4.14.2.jar:4.14.2]
at org.apache.bookkeeper.common.allocator.impl.ByteBufAllocatorImpl.newDirectBuffer(ByteBufAllocatorImpl.java:157) [org.apache.bookkeeper-bookkeeper-common-allocator-4.14.2.jar:4.14.2]
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188) [io.netty-netty-buffer-4.1.72.Final.jar:4.1.72.Final]
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179) [io.netty-netty-buffer-4.1.72.Final.jar:4.1.72.Final]
at io.netty.channel.unix.PreferredDirectByteBufAllocator.ioBuffer(PreferredDirectByteBufAllocator.java:53) [io.netty-netty-transport-native-unix-common-4.1.72.Final-linux-x86_64.jar:4.1.72.Final]
at io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:120) [io.netty-netty-transport-4.1.72.Final.jar:4.1.72.Final]
at io.netty.channel.epoll.EpollRecvByteAllocatorHandle.allocate(EpollRecvByteAllocatorHandle.java:75) [io.netty-netty-transport-classes-epoll-4.1.72.Final.jar:4.1.72.Final]
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:780) [io.netty-netty-transport-classes-epoll-4.1.72.Final.jar:4.1.72.Final]
at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe$1.run(AbstractEpollChannel.java:425) [io.netty-netty-transport-classes-epoll-4.1.72.Final.jar:4.1.72.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) [io.netty-netty-common-4.1.72.Final.jar:4.1.72.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469) [io.netty-netty-common-4.1.72.Final.jar:4.1.72.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) [io.netty-netty-transport-classes-epoll-4.1.72.Final.jar:4.1.72.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [io.netty-netty-common-4.1.72.Final.jar:4.1.72.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [io.netty-netty-common-4.1.72.Final.jar:4.1.72.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.72.Final.jar:4.1.72.Final]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
**To Reproduce**
Steps to reproduce the behavior:
1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error
**Expected behavior**
A clear and concise description of what you expected to happen.
**Screenshots**
If applicable, add screenshots to help explain your problem.
**Desktop (please complete the following information):**
- OS: [e.g. iOS]
**Additional context**
Add any other context about the problem here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #14468: [bookie-io-1-9] ERROR org.apache.bookkeeper.proto.BookieServer - Unable to allocate memory, exiting bookie
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on issue #14468:
URL: https://github.com/apache/pulsar/issues/14468#issuecomment-1067660905
@hangc0276 @zymap Could you please help check this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] jianke221 commented on issue #14468: [bookie-io-1-9] ERROR org.apache.bookkeeper.proto.BookieServer - Unable to allocate memory, exiting bookie
Posted by GitBox <gi...@apache.org>.
jianke221 commented on issue #14468:
URL: https://github.com/apache/pulsar/issues/14468#issuecomment-1050652918
armandxu @armandxu I are colleagues,Let me describe the phenomenon observed in the monitoring，We now test and find that directmemory will soar after the bookie node becomes readonly. After the ledger is expired and deleted, directmemory will stop soaring, but will not be released. After the next readonly, it will continue to soar. If it reaches the directmemory capacity before expiration and deletion, it will report an error and exit. What should we do in this case and what is the reason for the directmemory soaring
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] armandxu commented on issue #14468: [bookie-io-1-9] ERROR org.apache.bookkeeper.proto.BookieServer - Unable to allocate memory, exiting bookie
Posted by GitBox <gi...@apache.org>.
armandxu commented on issue #14468:
URL: https://github.com/apache/pulsar/issues/14468#issuecomment-1074942368
sorry for reply later,and ensemble size =2, write quorum=2, ack quorum=2,I can not reproduced this case,because i has new problem: broker can not found bookie,but bookie is ready:
peg)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] hangc0276 commented on issue #14468: [bookie-io-1-9] ERROR org.apache.bookkeeper.proto.BookieServer - Unable to allocate memory, exiting bookie
Posted by GitBox <gi...@apache.org>.
hangc0276 commented on issue #14468:
URL: https://github.com/apache/pulsar/issues/14468#issuecomment-1067692906
@jianke221 @armandxu Does this case can be reproduced?
Would you please share the steps and status of the BookKeeper cluster that you encountered this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] zymap commented on issue #14468: [bookie-io-1-9] ERROR org.apache.bookkeeper.proto.BookieServer - Unable to allocate memory, exiting bookie
Posted by GitBox <gi...@apache.org>.
zymap commented on issue #14468:
URL: https://github.com/apache/pulsar/issues/14468#issuecomment-1067722896
Also I want to know more configuration about the cluster. such as the ensemble size , write quorum, ack quorum
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] armandxu edited a comment on issue #14468: [bookie-io-1-9] ERROR org.apache.bookkeeper.proto.BookieServer - Unable to allocate memory, exiting bookie
Posted by GitBox <gi...@apache.org>.
armandxu edited a comment on issue #14468:
URL: https://github.com/apache/pulsar/issues/14468#issuecomment-1074942368
sorry for reply later,and ensemble size =2, write quorum=2, ack quorum=2,I can not reproduced this case,because i has new problem: broker can not found bookie,but bookie is ready:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org