You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2020/04/17 07:35:57 UTC
[GitHub] [pulsar] kaykillyou opened a new issue #6751:
org.apache.bookkeeper.proto.WriteEntryProcessor - Attempt to write to
fenced ledger
kaykillyou opened a new issue #6751: org.apache.bookkeeper.proto.WriteEntryProcessor - Attempt to write to fenced ledger
URL: https://github.com/apache/pulsar/issues/6751
**Describe the bug**
there a exception after restart bookie, and all machines in the cluster keep reporting errors as below:
2020-04-17 12:52:12.009 INFO [GarbageCollectorThread-11-1] org.apache.bookkeeper.bookie.GarbageCollectorThread - Enter minor compaction, suspendMinor false
2020-04-17 12:52:12.009 INFO [GarbageCollectorThread-11-1] org.apache.bookkeeper.bookie.GarbageCollectorThread - Do compaction to compact those files lower than 0.2
2020-04-17 12:52:12.040 INFO [db-storage-cleanup-10-1] org.apache.bookkeeper.bookie.storage.ldb.EntryLocationIndex - Deleting indexes for ledgers: [28960, 28725, 28965, 28726, 28727, 28967, 28280]
2020-04-17 12:52:12.172 INFO [db-storage-cleanup-10-1] org.apache.bookkeeper.bookie.storage.ldb.EntryLocationIndex - Deleted indexes for 100016 entries from 7 ledgers in 0.131 seconds
2020-04-17 12:54:19.134 INFO [GarbageCollectorThread-11-1] org.apache.bookkeeper.bookie.EntryLogCompactor - Removing entry log 214 after compaction
2020-04-17 12:54:19.200 INFO [GarbageCollectorThread-11-1] org.apache.bookkeeper.bookie.GarbageCollectorThread - Removing entry log metadata for 214
2020-04-17 12:54:19.200 INFO [GarbageCollectorThread-11-1] org.apache.bookkeeper.bookie.GarbageCollectorThread - Compaction: entry log usage buckets[10% 20% 30% 40% 50% 60% 70% 80% 90% 100%] = [0, 1, 0, 0, 0, 0, 0, 0, 0, 0]
2020-04-17 13:01:50.892 INFO [BookieJournal-3181] org.apache.bookkeeper.bookie.JournalChannel - Opening journal /data1/bookkeeper/journal/current/16f3c9ea3e4.txn
2020-04-17 13:02:12.041 INFO [SyncThread-7-1] org.apache.bookkeeper.bookie.Journal - garbage collected journal 16f3c9ea3de.txn
2020-04-17 13:07:12.036 INFO [db-storage-cleanup-10-1] org.apache.bookkeeper.bookie.storage.ldb.EntryLocationIndex - Deleting indexes for ledgers: [28976, 28978, 28981, 28982, 28983, 28973]
2020-04-17 13:07:12.374 INFO [db-storage-cleanup-10-1] org.apache.bookkeeper.bookie.storage.ldb.EntryLocationIndex - Deleted indexes for 250008 entries from 6 ledgers in 0.337 seconds
2020-04-17 13:22:12.036 INFO [db-storage-cleanup-10-1] org.apache.bookkeeper.bookie.storage.ldb.EntryLocationIndex - Deleting indexes for ledgers: [28980, 28984]
2020-04-17 13:22:12.380 INFO [db-storage-cleanup-10-1] org.apache.bookkeeper.bookie.storage.ldb.EntryLocationIndex - Deleted indexes for 100000 entries from 2 ledgers in 0.344 seconds
2020-04-17 13:26:48.176 INFO [bookie-io-1-12] org.apache.bookkeeper.proto.AuthHandler - Authentication success on server side
2020-04-17 13:26:48.176 INFO [bookie-io-1-12] org.apache.bookkeeper.proto.BookieRequestHandler - Channel connected [id: 0x410eafe2, L:/10.52.10.114:3181 - R:/10.52.10.115:47648]
2020-04-17 13:37:12.036 INFO [db-storage-cleanup-10-1] org.apache.bookkeeper.bookie.storage.ldb.EntryLocationIndex - Deleting indexes for ledgers: [28992, 28963, 28986]
2020-04-17 13:37:12.127 INFO [db-storage-cleanup-10-1] org.apache.bookkeeper.bookie.storage.ldb.EntryLocationIndex - Deleted indexes for 100984 entries from 3 ledgers in 0.09 seconds
2020-04-17 13:41:47.029 INFO [bookie-io-1-13] org.apache.bookkeeper.proto.AuthHandler - Authentication success on server side
2020-04-17 13:41:47.029 INFO [bookie-io-1-13] org.apache.bookkeeper.proto.BookieRequestHandler - Channel connected [id: 0x490c94fc, L:/10.52.10.114:3181 - R:/10.52.10.114:49494]
2020-04-17 13:41:47.385 INFO [bookie-io-1-14] org.apache.bookkeeper.proto.AuthHandler - Authentication success on server side
2020-04-17 13:41:47.385 INFO [bookie-io-1-14] org.apache.bookkeeper.proto.BookieRequestHandler - Channel connected [id: 0xceb71e0d, L:/10.52.10.114:3181 - R:/10.52.10.116:56866]
2020-04-17 13:41:47.386 INFO [bookie-io-1-15] org.apache.bookkeeper.proto.AuthHandler - Authentication success on server side
2020-04-17 13:41:47.386 INFO [bookie-io-1-15] org.apache.bookkeeper.proto.BookieRequestHandler - Channel connected [id: 0x2c0138fa, L:/10.52.10.114:3181 - R:/10.52.10.115:47792]
2020-04-17 13:41:47.388 INFO [bookie-io-1-16] org.apache.bookkeeper.proto.AuthHandler - Authentication success on server side
2020-04-17 13:41:47.388 INFO [bookie-io-1-16] org.apache.bookkeeper.proto.BookieRequestHandler - Channel connected [id: 0xcad36398, L:/10.52.10.114:3181 - R:/10.52.10.114:49502]
2020-04-17 13:42:11.346 ERROR [bookie-io-1-11] org.apache.bookkeeper.proto.WriteEntryProcessor - Attempt to write to fenced ledger
org.apache.bookkeeper.bookie.BookieException$LedgerFencedException: null
at org.apache.bookkeeper.bookie.BookieException.create(BookieException.java:57) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.bookie.Bookie.addEntry(Bookie.java:1374) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.proto.WriteEntryProcessor.processPacket(WriteEntryProcessor.java:79) [org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.proto.PacketProcessorBase.safeRun(PacketProcessorBase.java:85) [org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) [org.apache.bookkeeper-bookkeeper-common-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.proto.BookieRequestProcessor.processAddRequest(BookieRequestProcessor.java:616) [org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.proto.BookieRequestProcessor.processRequest(BookieRequestProcessor.java:365) [org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.proto.BookieRequestHandler.channelRead(BookieRequestHandler.java:80) [org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
at org.apache.bookkeeper.proto.AuthHandler$ServerSideHandler.channelRead(AuthHandler.java:89) [org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
at org.apache.bookkeeper.proto.BookieProtoEncoding$RequestDecoder.channelRead(BookieProtoEncoding.java:441) [org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:326) [io.netty-netty-codec-4.1.43.Final.jar:4.1.43.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:300) [io.netty-netty-codec-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1050) [io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]
**To Reproduce**
1. restart the bookie service incorrectly with root user
2. stop the bookie service and start service with normal user
3. an error will occur after the service is started
4. See error
**Expected behavior**
expected to solve this issue
**Screenshots**
**Desktop (please complete the following information):**
**Additional context**
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services