You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2020/06/30 14:54:54 UTC
[GitHub] [pulsar] KannarFr opened a new issue #7398: possible inconsistencies en broker local metadata cache
KannarFr opened a new issue #7398:
URL: https://github.com/apache/pulsar/issues/7398
**Describe the bug**
Not sure how to to describe it but:
Had the following stack looks like inconsistencies on a broker local metada that prevent namespace deletion. The namespace was considered empty by pulsar-admin-cli, but when I tried to run deletion one of the broker logged the following stack.
I just restarted the broker and everything worked well. I could run the deletion instantly.
```
Jun 30 14:42:29 yo-pulsar-c1-n3 pulsar-broker[10935]: 14:42:29.710 [bookkeeper-ml-workers-OrderedExecutor-6-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://user_7684cfc9-f54e-4e09-848c-1953af6e3e89/pulsar_c61f4274-4725-4e9f-8196-3cd7edcd77e5/test] Inactive topic deletion failed
Jun 30 14:42:29 yo-pulsar-c1-n3 pulsar-broker[10935]: java.util.concurrent.CompletionException: org.apache.pulsar.broker.service.BrokerServiceException$PersistenceException: org.apache.bookkeeper.mledger.ManagedLedgerException$MetaStoreException: java.util.concurrent.CompletionException: org.apache.pulsar.metadata.api.MetadataStoreException$NotFoundException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /managed-ledgers/user_7684cfc9-f54e-4e09-848c-1953af6e3e89/pulsar_c61f4274-4725-4e9f-8196-3cd7edcd77e5/persistent/test
Jun 30 14:42:29 yo-pulsar-c1-n3 pulsar-broker[10935]: at java.util.concurrent.CompletableFuture.encodeRelay(CompletableFuture.java:326) ~[?:1.8.0_192]
Jun 30 14:42:29 yo-pulsar-c1-n3 pulsar-broker[10935]: at java.util.concurrent.CompletableFuture.completeRelay(CompletableFuture.java:338) ~[?:1.8.0_192]
Jun 30 14:42:29 yo-pulsar-c1-n3 pulsar-broker[10935]: at java.util.concurrent.CompletableFuture.uniRelay(CompletableFuture.java:911) ~[?:1.8.0_192]
Jun 30 14:42:29 yo-pulsar-c1-n3 pulsar-broker[10935]: at java.util.concurrent.CompletableFuture$UniRelay.tryFire(CompletableFuture.java:899) ~[?:1.8.0_192]
Jun 30 14:42:29 yo-pulsar-c1-n3 pulsar-broker[10935]: at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_192]
Jun 30 14:42:29 yo-pulsar-c1-n3 pulsar-broker[10935]: at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_192]
Jun 30 14:42:29 yo-pulsar-c1-n3 pulsar-broker[10935]: at org.apache.pulsar.broker.service.persistent.PersistentTopic$4.deleteLedgerFailed(PersistentTopic.java:914) ~[org.apache.pulsar-pulsar-broker-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
Jun 30 14:42:29 yo-pulsar-c1-n3 pulsar-broker[10935]: at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl$18.operationFailed(ManagedLedgerImpl.java:2333) ~[org.apache.pulsar-managed-ledger-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
Jun 30 14:42:29 yo-pulsar-c1-n3 pulsar-broker[10935]: at org.apache.bookkeeper.mledger.impl.MetaStoreImpl.lambda$null$21(MetaStoreImpl.java:218) ~[org.apache.pulsar-managed-ledger-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
Jun 30 14:42:29 yo-pulsar-c1-n3 pulsar-broker[10935]: at org.apache.bookkeeper.util.SafeRunnable$1.safeRun(SafeRunnable.java:43) [org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
Jun 30 14:42:29 yo-pulsar-c1-n3 pulsar-broker[10935]: at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) [org.apache.bookkeeper-bookkeeper-common-4.10.0.jar:4.10.0]
Jun 30 14:42:29 yo-pulsar-c1-n3 pulsar-broker[10935]: at org.apache.bookkeeper.common.util.OrderedExecutor$TimedRunnable.run(OrderedExecutor.java:203) [org.apache.bookkeeper-bookkeeper-common-4.10.0.jar:4.10.0]
Jun 30 14:42:29 yo-pulsar-c1-n3 pulsar-broker[10935]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_192]
Jun 30 14:42:29 yo-pulsar-c1-n3 pulsar-broker[10935]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_192]
Jun 30 14:42:29 yo-pulsar-c1-n3 pulsar-broker[10935]: at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.48.Final.jar:4.1.48.Final]
Jun 30 14:42:29 yo-pulsar-c1-n3 pulsar-broker[10935]: at java.lang.Thread.run(Thread.java:748) [?:1.8.0_192]
Jun 30 14:42:29 yo-pulsar-c1-n3 pulsar-broker[10935]: Caused by: org.apache.pulsar.broker.service.BrokerServiceException$PersistenceException: org.apache.bookkeeper.mledger.ManagedLedgerException$MetaStoreException: java.util.concurrent.CompletionException: org.apache.pulsar.metadata.api.MetadataStoreException$NotFoundException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /managed-ledgers/user_7684cfc9-f54e-4e09-848c-1953af6e3e89/pulsar_c61f4274-4725-4e9f-8196-3cd7edcd77e5/persistent/test
Jun 30 14:42:29 yo-pulsar-c1-n3 pulsar-broker[10935]: ... 10 more
Jun 30 14:42:29 yo-pulsar-c1-n3 pulsar-broker[10935]: Caused by: org.apache.bookkeeper.mledger.ManagedLedgerException$MetaStoreException: java.util.concurrent.CompletionException: org.apache.pulsar.metadata.api.MetadataStoreException$NotFoundException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /managed-ledgers/user_7684cfc9-f54e-4e09-848c-1953af6e3e89/pulsar_c61f4274-4725-4e9f-8196-3cd7edcd77e5/persistent/test
Jun 30 14:42:29 yo-pulsar-c1-n3 pulsar-broker[10935]: Caused by: java.util.concurrent.CompletionException: org.apache.pulsar.metadata.api.MetadataStoreException$NotFoundException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /managed-ledgers/user_7684cfc9-f54e-4e09-848c-1953af6e3e89/pulsar_c61f4274-4725-4e9f-8196-3cd7edcd77e5/persistent/test
Jun 30 14:42:29 yo-pulsar-c1-n3 pulsar-broker[10935]: at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) ~[?:1.8.0_192]
Jun 30 14:42:29 yo-pulsar-c1-n3 pulsar-broker[10935]: at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) ~[?:1.8.0_192]
Jun 30 14:42:29 yo-pulsar-c1-n3 pulsar-broker[10935]: at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:647) ~[?:1.8.0_192]
Jun 30 14:42:29 yo-pulsar-c1-n3 pulsar-broker[10935]: at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:632) ~[?:1.8.0_192]
Jun 30 14:42:29 yo-pulsar-c1-n3 pulsar-broker[10935]: at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_192]
Jun 30 14:42:29 yo-pulsar-c1-n3 pulsar-broker[10935]: at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_192]
Jun 30 14:42:29 yo-pulsar-c1-n3 pulsar-broker[10935]: at org.apache.pulsar.metadata.impl.zookeeper.ZKMetadataStore.lambda$null$16(ZKMetadataStore.java:238) ~[org.apache.pulsar-pulsar-metadata-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
Jun 30 14:42:29 yo-pulsar-c1-n3 pulsar-broker[10935]: ... 4 more
Jun 30 14:42:29 yo-pulsar-c1-n3 pulsar-broker[10935]: Caused by: org.apache.pulsar.metadata.api.MetadataStoreException$NotFoundException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /managed-ledgers/user_7684cfc9-f54e-4e09-848c-1953af6e3e89/pulsar_c61f4274-4725-4e9f-8196-3cd7edcd77e5/persistent/test
Jun 30 14:42:29 yo-pulsar-c1-n3 pulsar-broker[10935]: at org.apache.pulsar.metadata.impl.zookeeper.ZKMetadataStore.getException(ZKMetadataStore.java:268) ~[org.apache.pulsar-pulsar-metadata-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
Jun 30 14:42:29 yo-pulsar-c1-n3 pulsar-broker[10935]: at org.apache.pulsar.metadata.impl.zookeeper.ZKMetadataStore.lambda$null$16(ZKMetadataStore.java:238) ~[org.apache.pulsar-pulsar-metadata-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
Jun 30 14:42:29 yo-pulsar-c1-n3 pulsar-broker[10935]: ... 4 more
Jun 30 14:42:29 yo-pulsar-c1-n3 pulsar-broker[10935]: Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /managed-ledgers/user_7684cfc9-f54e-4e09-848c-1953af6e3e89/pulsar_c61f4274-4725-4e9f-8196-3cd7edcd77e5/persistent/test
Jun 30 14:42:29 yo-pulsar-c1-n3 pulsar-broker[10935]: at org.apache.zookeeper.KeeperException.create(KeeperException.java:118) ~[org.apache.pulsar-pulsar-zookeeper-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
Jun 30 14:42:29 yo-pulsar-c1-n3 pulsar-broker[10935]: at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[org.apache.pulsar-pulsar-zookeeper-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
Jun 30 14:42:29 yo-pulsar-c1-n3 pulsar-broker[10935]: at org.apache.pulsar.metadata.impl.zookeeper.ZKMetadataStore.getException(ZKMetadataStore.java:262) ~[org.apache.pulsar-pulsar-metadata-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
Jun 30 14:42:29 yo-pulsar-c1-n3 pulsar-broker[10935]: at org.apache.pulsar.metadata.impl.zookeeper.ZKMetadataStore.lambda$null$16(ZKMetadataStore.java:238) ~[org.apache.pulsar-pulsar-metadata-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
Jun 30 14:42:29 yo-pulsar-c1-n3 pulsar-broker[10935]: ... 4 more
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] jiazhai commented on issue #7398: possible inconsistencies en broker local metadata cache
Posted by GitBox <gi...@apache.org>.
jiazhai commented on issue #7398:
URL: https://github.com/apache/pulsar/issues/7398#issuecomment-652738200
Thanks @KannarFr for open this issue. Are there steps to reproduce this issue?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] KannarFr commented on issue #7398: possible inconsistencies en broker local metadata cache
Posted by GitBox <gi...@apache.org>.
KannarFr commented on issue #7398:
URL: https://github.com/apache/pulsar/issues/7398#issuecomment-652932518
@jiazhai I have no reproductible case :/. It occurs on our production randomly on a namespace deletion.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] shulaoh commented on issue #7398: possible inconsistencies en broker local metadata cache
Posted by GitBox <gi...@apache.org>.
shulaoh commented on issue #7398:
URL: https://github.com/apache/pulsar/issues/7398#issuecomment-916572622
Just wondering why this issue is closed, is it harmless or fixed indirectly?
I hit this issue by using pulsar as flink source.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] KannarFr closed issue #7398: possible inconsistencies en broker local metadata cache
Posted by GitBox <gi...@apache.org>.
KannarFr closed issue #7398:
URL: https://github.com/apache/pulsar/issues/7398
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org