You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/04/22 05:41:13 UTC
[GitHub] [pulsar] 346359827 opened a new issue, #15267: No such ledger exists on Metadata Server
346359827 opened a new issue, #15267:
URL: https://github.com/apache/pulsar/issues/15267
**Describe the bug**
No such ledger exists on Metadata Server
00:01:37.560 [pulsar-io-4-51] WARN org.apache.bookkeeper.mledger.impl.ManagedCursorImpl - [b2d/rcp-new/persistent/rcp-normal%23ms
g-topic%23keyed%23t1lJtAuTsZ5GVmUmczFjf3-partition-1] [rcp-normal-queue-subscription] Error while updating individualDeletedMessag
es [Mark deleting an already mark-deleted position]
java.lang.IllegalArgumentException: Mark deleting an already mark-deleted position
at org.apache.bookkeeper.mledger.impl.ManagedCursorImpl.setAcknowledgedPosition(ManagedCursorImpl.java:1582) ~[org.apache.
pulsar-managed-ledger-2.8.2.jar:2.8.2]
at org.apache.bookkeeper.mledger.impl.ManagedCursorImpl.asyncDelete(ManagedCursorImpl.java:2016) [org.apache.pulsar-manage
d-ledger-2.8.2.jar:2.8.2]
at org.apache.pulsar.broker.service.persistent.PersistentSubscription.acknowledgeMessage(PersistentSubscription.java:393)
[org.apache.pulsar-pulsar-broker-2.8.2.jar:2.8.2]
at org.apache.pulsar.broker.service.Consumer.individualAckNormal(Consumer.java:410) [org.apache.pulsar-pulsar-broker-2.8.2
.jar:2.8.2]
at org.apache.pulsar.broker.service.Consumer.messageAcked(Consumer.java:376) [org.apache.pulsar-pulsar-broker-2.8.2.jar:2.
8.2]
at org.apache.pulsar.broker.service.ServerCnx.handleAck(ServerCnx.java:1416) [org.apache.pulsar-pulsar-broker-2.8.2.jar:2.
8.2]
at org.apache.pulsar.common.protocol.PulsarDecoder.channelRead(PulsarDecoder.java:144) [org.apache.pulsar-pulsar-common-2.
8.2.jar:2.8.2]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [io.netty-nett
y-transport-4.1.72.Final.jar:4.1.72.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [io.netty-nett
y-transport-4.1.72.Final.jar:4.1.72.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [io.netty-netty-
transport-4.1.72.Final.jar:4.1.72.Final]
at io.netty.handler.flow.FlowControlHandler.dequeue(FlowControlHandler.java:200) [io.netty-netty-handler-4.1.72.Final.jar:
4.1.72.Final]
at io.netty.handler.flow.FlowControlHandler.channelRead(FlowControlHandler.java:162) [io.netty-netty-handler-4.1.72.Final.
jar:4.1.72.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [io.netty-nett
y-transport-4.1.72.Final.jar:4.1.72.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [io.netty-nett
y-transport-4.1.72.Final.jar:4.1.72.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [io.netty-netty-
transport-4.1.72.Final.jar:4.1.72.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) [io.netty-netty-codec-4.1.72
.Final.jar:4.1.72.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:311) [io.netty-netty-codec-4.1.72
.Final.jar:4.1.72.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:432) [io.netty-netty-codec-4.1.72.Fina
l.jar:4.1.72.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) [io.netty-netty-codec-4.1.72.Fin
al.jar:4.1.72.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [io.netty-nett
y-transport-4.1.72.Final.jar:4.1.72.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [io.netty-nett
y-transport-4.1.72.Final.jar:4.1.72.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [io.netty-netty-
transport-4.1.72.Final.jar:4.1.72.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [io.netty-netty-trans
port-4.1.72.Final.jar:4.1.72.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [io.netty-nett
y-transport-4.1.72.Final.jar:4.1.72.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [io.netty-nett
y-transport-4.1.72.Final.jar:4.1.72.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [io.netty-netty-transport-4.1.
72.Final.jar:4.1.72.Final]
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795) [
io.netty-netty-transport-classes-epoll-4.1.72.Final.jar:4.1.72.Final]
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480) [io.netty-netty-transport-classes-epoll-4.1
.72.Final.jar:4.1.72.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) [io.netty-netty-transport-classes-epoll-4.1.72.Final
.jar:4.1.72.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [io.netty-netty-common-4.1
.72.Final.jar:4.1.72.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [io.netty-netty-common-4.1.72.Final.jar:4.1.7
2.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.72.Fin
al.jar:4.1.72.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
00:01:37.561 [pulsar-io-4-51] WARN org.apache.pulsar.broker.service.persistent.PersistentSubscription - [persistent://b2d/rcp-new
/rcp-normal#msg-topic#keyed#t1lJtAuTsZ5GVmUmczFjf3-partition-1][rcp-normal-queue-subscription] Failed to delete message at 286371:
-1: org.apache.bookkeeper.mledger.ManagedLedgerException: java.lang.IllegalArgumentException: Mark deleting an already mark-delete
d position
00:01:46.653 [BookKeeperClientWorker-OrderedExecutor-18-0] ERROR org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [b2d/rcp-n
ew/persistent/rcp-normal%23broadcast-topic] Error opening ledger for reading at position 310871:0 - org.apache.bookkeeper.mledger.
ManagedLedgerException$NonRecoverableLedgerException: No such ledger exists on Metadata Server
00:01:46.653 [BookKeeperClientWorker-OrderedExecutor-18-0] WARN org.apache.bookkeeper.mledger.impl.OpReadEntry - [b2d/rcp-new/per
sistent/rcp-normal%23broadcast-topic][rcp-normal-queue-subscription%23priority-queue-broadcast-subscription%236da683ae-2317-4541-b
378-e33aac9e7241] read failed from ledger at position:310871:0
org.apache.bookkeeper.mledger.ManagedLedgerException$NonRecoverableLedgerException: No such ledger exists on Metadata Server
00:01:46.656 [broker-topic-workers-OrderedScheduler-0-0] ERROR org.apache.pulsar.broker.service.persistent.PersistentDispatcherSin
gleActiveConsumer - [persistent://b2d/rcp-new/rcp-normal#broadcast-topic / rcp-normal-queue-subscription#priority-queue-broadcast-
subscription#6da683ae-2317-4541-b378-e33aac9e7241-Consumer{subscription=PersistentSubscription{topic=persistent://b2d/rcp-new/rcp-
normal#broadcast-topic, name=rcp-normal-queue-subscription#priority-queue-broadcast-subscription#6da683ae-2317-4541-b378-e33aac9e7
241}, consumerId=0, consumerName=a9e48, address=/192.168.0.25:39661}] Error reading entries at 310871:0 : No such ledger exists on
Metadata Server - Retrying to read in 56.778 seconds
00:01:49.081 [ForkJoinPool.commonPool-worker-6] ERROR org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://
pulsar/hz-gt-pulsar-cluster/192.168.0.92:8080/healthcheck] Error deleting topic
java.util.concurrent.CompletionException: org.apache.pulsar.broker.service.schema.exceptions.SchemaException: No such ledger exist
s on Metadata Server - ledger=672948 - operation=Failed to open ledger
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) ~[?:1.8.0_162]
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) ~[?:1.8.0_162]
at java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:943) ~[?:1.8.0_162]
at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:926) ~[?:1.8.0_162]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_162]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_162]
at org.apache.pulsar.broker.service.schema.BookkeeperSchemaStorage.lambda$openLedger$40(BookkeeperSchemaStorage.java:594)
~[org.apache.pulsar-pulsar-broker-2.8.2.jar:2.8.2]
at org.apache.bookkeeper.client.LedgerOpenOp.openComplete(LedgerOpenOp.java:248) ~[org.apache.bookkeeper-bookkeeper-server
-4.14.3.jar:4.14.3]
at org.apache.bookkeeper.client.LedgerOpenOp.lambda$initiate$0(LedgerOpenOp.java:117) ~[org.apache.bookkeeper-bookkeeper-s
erver-4.14.3.jar:4.14.3]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_162]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[?:1.8.0_162]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_162]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_162]
at org.apache.bookkeeper.meta.AbstractZkLedgerManager$3.processResult(AbstractZkLedgerManager.java:443) ~[org.apache.bookk
eeper-bookkeeper-server-4.14.3.jar:4.14.3]
at org.apache.bookkeeper.zookeeper.ZooKeeperClient$19$1.processResult(ZooKeeperClient.java:997) ~[org.apache.bookkeeper-bo
okkeeper-server-4.14.3.jar:4.14.3]
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:646) ~[org.apache.zookeeper-zookeeper-3.6.3.ja
r:3.6.3]
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:563) ~[org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3]
Caused by: org.apache.pulsar.broker.service.schema.exceptions.SchemaException: No such ledger exists on Metadata Server - ledger=
672948 - operation=Failed to open ledger
at org.apache.pulsar.broker.service.schema.BookkeeperSchemaStorage.bkException(BookkeeperSchemaStorage.java:700) ~[org.apa
che.pulsar-pulsar-broker-2.8.2.jar:2.8.2]
... 11 more
00:01:49.083 [ForkJoinPool.commonPool-worker-6] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://
pulsar/hz-gt-pulsar-cluster/192.168.0.92:8080/healthcheck] Inactive topic deletion failed
java.util.concurrent.CompletionException: org.apache.pulsar.broker.service.schema.exceptions.SchemaException: No such ledger exist
s on Metadata Server - ledger=672948 - operation=Failed to open ledger
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) ~[?:1.8.0_162]
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) ~[?:1.8.0_162]
at java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:943) ~[?:1.8.0_162]
at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:926) ~[?:1.8.0_162]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_162]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_162]
at org.apache.pulsar.broker.service.schema.BookkeeperSchemaStorage.lambda$openLedger$40(BookkeeperSchemaStorage.java:594)
~[org.apache.pulsar-pulsar-broker-2.8.2.jar:2.8.2]
at org.apache.bookkeeper.client.LedgerOpenOp.openComplete(LedgerOpenOp.java:248) ~[org.apache.bookkeeper-bookkeeper-server
-4.14.3.jar:4.14.3]
at org.apache.bookkeeper.client.LedgerOpenOp.lambda$initiate$0(LedgerOpenOp.java:117) ~[org.apache.bookkeeper-bookkeeper-s
erver-4.14.3.jar:4.14.3]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_162]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[?:1.8.0_162]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_162]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_162]
at org.apache.bookkeeper.meta.AbstractZkLedgerManager$3.processResult(AbstractZkLedgerManager.java:443) ~[org.apache.bookk
eeper-bookkeeper-server-4.14.3.jar:4.14.3]
at org.apache.bookkeeper.zookeeper.ZooKeeperClient$19$1.processResult(ZooKeeperClient.java:997) ~[org.apache.bookkeeper-bo
okkeeper-server-4.14.3.jar:4.14.3]
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:646) ~[org.apache.zookeeper-zookeeper-3.6.3.ja
r:3.6.3]
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:563) ~[org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3]
Caused by: org.apache.pulsar.broker.service.schema.exceptions.SchemaException: No such ledger exists on Metadata Server - ledger=
672948 - operation=Failed to open ledger
at org.apache.pulsar.broker.service.schema.BookkeeperSchemaStorage.bkException(BookkeeperSchemaStorage.java:700) ~[org.apa
che.pulsar-pulsar-broker-2.8.2.jar:2.8.2]
... 11 more
00:01:52.173 [pulsar-io-4-38] INFO org.apache.pulsar.broker.service.ServerCnx - New connection from /192.168.0.94:54518
00:01:52.173 [pulsar-io-4-38] INFO org.apache.pulsar.broker.service.ServerCnx - Closed connection from /192.168.0.94:54518
00:01:57.679 [BookKeeperClientWorker-OrderedExecutor-27-0] INFO org.apache.bookkeeper.mledger.impl.OpAddEntry - [b2d/rcp-new/pers
istent/rcp-normal%23msg-topic%23keyed%23MHoCdPYZFP822t1xrTapN2-partition-1] Closing ledger 341816 for being full
00:01:57.699 [main-EventThread] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [b2d/rcp-new/persistent/rcp-normal%23
msg-topic%23keyed%23MHoCdPYZFP822t1xrTapN2-partition-1] Creating a new ledger
version 2.8.2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on issue #15267: No such ledger exists on Metadata Server
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #15267:
URL: https://github.com/apache/pulsar/issues/15267#issuecomment-1135324459
The issue had no activity for 30 days, mark with Stale label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] tqdkhoa commented on issue #15267: No such ledger exists on Metadata Server
Posted by "tqdkhoa (via GitHub)" <gi...@apache.org>.
tqdkhoa commented on issue #15267:
URL: https://github.com/apache/pulsar/issues/15267#issuecomment-1737741691
Hi, we also get the same issue on the public tenant, and there is no workaround solution. So, if anyone has resolved it, please share your solution, much appreciated because we stuck here for a while
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyuanpeng commented on issue #15267: No such ledger exists on Metadata Server
Posted by GitBox <gi...@apache.org>.
liangyuanpeng commented on issue #15267:
URL: https://github.com/apache/pulsar/issues/15267#issuecomment-1107456168
It is posibble we reproduce it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on issue #15267: No such ledger exists on Metadata Server
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #15267:
URL: https://github.com/apache/pulsar/issues/15267#issuecomment-1163847080
The issue had no activity for 30 days, mark with Stale label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gjanardh commented on issue #15267: No such ledger exists on Metadata Server
Posted by GitBox <gi...@apache.org>.
gjanardh commented on issue #15267:
URL: https://github.com/apache/pulsar/issues/15267#issuecomment-1178237827
We had a similar issue in our DEV environment. Deleting a topic when the BK was faulty, could have caused it in our environment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] enzo-dechaene commented on issue #15267: No such ledger exists on Metadata Server
Posted by "enzo-dechaene (via GitHub)" <gi...@apache.org>.
enzo-dechaene commented on issue #15267:
URL: https://github.com/apache/pulsar/issues/15267#issuecomment-1643539862
We had a similar issue. We cannot delete the topic, how did you fix it? @gjanardh @346359827
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on issue #15267: No such ledger exists on Metadata Server
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #15267:
URL: https://github.com/apache/pulsar/issues/15267#issuecomment-1207316376
The issue had no activity for 30 days, mark with Stale label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] No such ledger exists on Metadata Server [pulsar]
Posted by "Technoboy- (via GitHub)" <gi...@apache.org>.
Technoboy- closed issue #15267: No such ledger exists on Metadata Server
URL: https://github.com/apache/pulsar/issues/15267
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org