You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/10/31 15:42:24 UTC
[GitHub] [pulsar] zbentley opened a new issue #12551: Attempting to delete topics shortly after creation/deletion requests fails with a 500 error of "MetadataNotFoundException: Managed ledger not found"
zbentley opened a new issue #12551:
URL: https://github.com/apache/pulsar/issues/12551
**Describe the bug**
When testing, I often tear down tenants/namespaces/topics and re-create those same tenants/namespaces/topics immediately thereafter to reset test state.
Sometimes (but not always) when I create a tenant/namespace/topic and then immediately request their deletion, I get a 500 error back from the management API, which correlates to the below stacktrace on the broker.
This should not happen; management API deletion operations should only return when deletion of all related resources has completed.
**To Reproduce**
This doesn't reproduce 100% of the time. It seems likely to happen more often when it is done on a brand new broker cluster that hasn't ever hosted topics/non-default tenants.
1. Create a tenant.
2. Create a namespace in that tenant
3. Create a persistent, partitioned topic in that namespace.
4. Create a subscription in that topic and connect at least one consumer to that subscription (no messages need to be produced or consumed).
5. Immediately thereafter, attempt to delete the topic/namespace/tenant.
**Expected behavior**
1. No errors occur.
2. This is probably more important: all operations related to deletion/creation of topics/namespaces/tenants conclude *before* the management API returns success codes to the user. This bug and the other similar ones I filed (see github links below) all seem to arise from CRUD operations with the management API being asynchronous: i.e. when I create/delete a tenant/topic/namespace, the actual side effects of that creation or deletion (e.g. adding/removing ledgers in BookKeeper, updating metadata in ZK) occur later, not during the API post. Not only is that bound to cause bugs like this, but it's also not what users expect; I would be happy to wait seconds or minutes for management API operations to complete in exchange for knowing that when they successfully complete that the thing I requested has actually been done.
**Environment**
- Kubernetes on EKS 1.18. StreamNative Platform Pulsar Broker 2.1.8.5.
**Stacktrace**
```
pulsar-blt-sn-platform-broker-2 java.util.concurrent.ExecutionException: org.apache.bookkeeper.mledger.ManagedLedgerException$MetadataNotFoundException: Managed ledger not found │
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395) ~[?:?] │
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999) ~[?:?] │
at org.apache.pulsar.broker.admin.impl.PersistentTopicsBase.internalDeleteTopicForcefully(PersistentTopicsBase.java:304) ~[io.streamnative-pulsar-broker-2.8.1.5.jar:2.8.1.5] │
at org.apache.pulsar.broker.admin.impl.PersistentTopicsBase.internalDeleteTopic(PersistentTopicsBase.java:946) ~[io.streamnative-pulsar-broker-2.8.1.5.jar:2.8.1.5] │
at org.apache.pulsar.broker.admin.v2.PersistentTopics.deleteTopic(PersistentTopics.java:889) ~[io.streamnative-pulsar-broker-2.8.1.5.jar:2.8.1.5] │
at jdk.internal.reflect.GeneratedMethodAccessor397.invoke(Unknown Source) ~[?:?] │
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] │
at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?] │
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[org.glassfish.jersey.core-jerse │
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124) ~[org.glassfish.jersey.core-jersey-server-2.34 │
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167) ~[org.glassfish.jersey.core-jersey-server-2.3 │
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$VoidOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:159) ~[org.glassfish.jersey.cor │
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79) ~[org.glassfish.jersey.core-jersey-server-2. │
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:475) ~[org.glassfish.jersey.core-jersey-server-2.34.jar:?] │
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:397) ~[org.glassfish.jersey.core-jersey-server-2.34.jar:?] │
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81) ~[org.glassfish.jersey.core-jersey-server-2.34.jar:?] │
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:255) ~[org.glassfish.jersey.core-jersey-server-2.34.jar:?] │
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?] │
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?] │
at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?] │
at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?] │
at org.glassfish.jersey.internal.Errors.process(Errors.java:244) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?] │
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?] │
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:234) ~[org.glassfish.jersey.core-jersey-server-2.34.jar:?] │
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680) ~[org.glassfish.jersey.core-jersey-server-2.34.jar:?] │
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394) ~[org.glassfish.jersey.containers-jersey-container-servlet-core-2.34.jar:?] │
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346) ~[org.glassfish.jersey.containers-jersey-container-servlet-core-2.34.jar:?] │
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366) ~[org.glassfish.jersey.containers-jersey-container-servlet-core-2.34.jar:?] │
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319) ~[org.glassfish.jersey.containers-jersey-container-servlet-core-2.34.jar:?] │
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205) ~[org.glassfish.jersey.containers-jersey-container-servlet-core-2.34.jar:?] │
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) ~[org.eclipse.jetty-jetty-servlet-9.4.43.v20210629.jar:9.4.43.v20210629] │
at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626) ~[org.eclipse.jetty-jetty-servlet-9.4.43.v20210629.jar:9.4.43.v20210629] │
at org.apache.pulsar.broker.web.ResponseHandlerFilter.doFilter(ResponseHandlerFilter.java:67) ~[io.streamnative-pulsar-broker-2.8.1.5.jar:2.8.1.5] │
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[org.eclipse.jetty-jetty-servlet-9.4.43.v20210629.jar:9.4.43.v20210629] │
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) ~[org.eclipse.jetty-jetty-servlet-9.4.43.v20210629.jar:9.4.43.v20210629] │
at org.apache.pulsar.broker.intercept.BrokerInterceptor.onFilter(BrokerInterceptor.java:88) ~[io.streamnative-pulsar-broker-2.8.1.5.jar:2.8.1.5] │
at org.apache.pulsar.broker.web.ProcessHandlerFilter.doFilter(ProcessHandlerFilter.java:49) ~[io.streamnative-pulsar-broker-2.8.1.5.jar:2.8.1.5]
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[org.eclipse.jetty-jetty-servlet-9.4.43.v20210629.jar:9.4.43.v20210629] │
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) ~[org.eclipse.jetty-jetty-servlet-9.4.43.v20210629.jar:9.4.43.v20210629] │
at org.apache.pulsar.broker.web.PreInterceptFilter.doFilter(PreInterceptFilter.java:70) ~[io.streamnative-pulsar-broker-2.8.1.5.jar:2.8.1.5] │
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[org.eclipse.jetty-jetty-servlet-9.4.43.v20210629.jar:9.4.43.v20210629] │
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) ~[org.eclipse.jetty-jetty-servlet-9.4.43.v20210629.jar:9.4.43.v20210629] │
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548) ~[org.eclipse.jetty-jetty-servlet-9.4.43.v20210629.jar:9.4.43.v20210629] │
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629] │
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) ~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629] │
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629] │
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434) ~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629] │
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) ~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629] │
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501) ~[org.eclipse.jetty-jetty-servlet-9.4.43.v20210629.jar:9.4.43.v20210629] │
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) ~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629] │
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) ~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629] │
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349) ~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629] │
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629] │
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:234) ~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629] │
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146) ~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629] │
at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:179) ~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629] │
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629] │
at org.eclipse.jetty.server.Server.handle(Server.java:516) ~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629] │
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388) ~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629] │
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633) [org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629] │
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380) [org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629] │
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) [org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629] │
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) [org.eclipse.jetty-jetty-io-9.4.43.v20210629.jar:9.4.43.v20210629] │
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) [org.eclipse.jetty-jetty-io-9.4.43.v20210629.jar:9.4.43.v20210629] │
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) [org.eclipse.jetty-jetty-io-9.4.43.v20210629.jar:9.4.43.v20210629] │
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) [org.eclipse.jetty-jetty-util-9.4.43.v20210629.jar:9.4.43.v20210629] │
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) [org.eclipse.jetty-jetty-util-9.4.43.v20210629.jar:9.4.43.v20210629] │
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) [org.eclipse.jetty-jetty-util-9.4.43.v20210629.jar:9.4.43.v20210629] │
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) [org.eclipse.jetty-jetty-util-9.4.43.v20210629.jar:9.4.43.v20210629] │
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:386) [org.eclipse.jetty-jetty-util-9.4.43.v20210629.jar:9.4.43.v20210629] │
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] │
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] │
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.68.Final.jar:4.1.68.Final] │
at java.lang.Thread.run(Thread.java:829) [?:?] │
│ pulsar-blt-sn-platform-broker-2 Caused by: org.apache.bookkeeper.mledger.ManagedLedgerException$MetadataNotFoundException: Managed ledger not found
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] merlimat commented on issue #12551: Attempting to delete topics shortly after creation/deletion requests fails with a 500 error of "MetadataNotFoundException: Managed ledger not found"
Posted by GitBox <gi...@apache.org>.
merlimat commented on issue #12551:
URL: https://github.com/apache/pulsar/issues/12551#issuecomment-955732165
> This is probably more important: all operations related to deletion/creation of topics/namespaces/tenants conclude before the management API returns success codes to the user.
The operations are always acked after the writes are completed. What typically happens is that the 2nd request goes to a different broker which has not yet received a cache invalidation signal (eg: zk watch).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] zbentley commented on issue #12551: Attempting to delete topics shortly after creation/deletion requests fails with a 500 error of "MetadataNotFoundException: Managed ledger not found"
Posted by GitBox <gi...@apache.org>.
zbentley commented on issue #12551:
URL: https://github.com/apache/pulsar/issues/12551#issuecomment-955750920
Though not all of these are just due to metadata being out of sync @merlimat; https://github.com/apache/pulsar/issues/12556 appears to stick around forever (I gave it tens of minutes before tearing down the cluster, at least, and it didn't go away on its own).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] zbentley commented on issue #12551: Attempting to delete topics shortly after creation/deletion requests fails with a 500 error of "MetadataNotFoundException: Managed ledger not found"
Posted by GitBox <gi...@apache.org>.
zbentley commented on issue #12551:
URL: https://github.com/apache/pulsar/issues/12551#issuecomment-997283295
@Technoboy- that doesn't seem accurate; this error manifests as a 500 error returned to clients. The deletion attempt used in my initial reproduction of this bug did use `force=true`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on issue #12551: Attempting to delete topics shortly after creation/deletion requests fails with a 500 error of "MetadataNotFoundException: Managed ledger not found"
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #12551:
URL: https://github.com/apache/pulsar/issues/12551#issuecomment-1064694737
The issue had no activity for 30 days, mark with Stale label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michaeljmarshall commented on issue #12551: Attempting to delete topics shortly after creation/deletion requests fails with a 500 error of "MetadataNotFoundException: Managed ledger not found"
Posted by GitBox <gi...@apache.org>.
michaeljmarshall commented on issue #12551:
URL: https://github.com/apache/pulsar/issues/12551#issuecomment-1032909979
It's possible that the issue here is a result of a missed cache invalidation: https://github.com/apache/pulsar/pull/14177. I'm not sure which version of Pulsar `Kubernetes on EKS 1.18. StreamNative Platform Pulsar Broker 2.1.8.5.` this is, but the bug I fix in #14177 definitely impacts 2.8.2 and leads the broker to cache a child node for at least 5 minutes under certain conditions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Technoboy- commented on issue #12551: Attempting to delete topics shortly after creation/deletion requests fails with a 500 error of "MetadataNotFoundException: Managed ledger not found"
Posted by GitBox <gi...@apache.org>.
Technoboy- commented on issue #12551:
URL: https://github.com/apache/pulsar/issues/12551#issuecomment-988453281
For this issue, the broker will only print the log containing `MetadataNotFoundException: Managed ledger not found`, but won't throw any exception.
That means: when deleting the topic forcibly, the broker may print this log, but the response is 200.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] zbentley edited a comment on issue #12551: Attempting to delete topics shortly after creation/deletion requests fails with a 500 error of "MetadataNotFoundException: Managed ledger not found"
Posted by GitBox <gi...@apache.org>.
zbentley edited a comment on issue #12551:
URL: https://github.com/apache/pulsar/issues/12551#issuecomment-955748576
The following issues were all observed in response to similar testing:
https://github.com/apache/pulsar/issues/12557
https://github.com/apache/pulsar/issues/12556
https://github.com/apache/pulsar/issues/12555
https://github.com/apache/pulsar/issues/12554
https://github.com/apache/pulsar/issues/12553
https://github.com/apache/pulsar/issues/12552
https://github.com/apache/pulsar/issues/12551
The condition that caused these issues to occur appears to be interaction with various pulsar entities (e.g. creating/deleting things in the management API, or attempting to create consumers) *immediately after those entities were created* or *immediately after entities with the same name were deleted*.
I think the number of issues observed speaks to a defect in the management API functionality in general. Considering the severity of these issues (in many cases it is possible to force a topic/namespace into a permanently corrupted state), I hope a resolution can be found for the general/common root cause rather than fixing individual bug-inducing conditions.
I suspect that the common root cause is that many management API operations are asynchronous that should not be.
Ideally, the resolution of all of these issues would be the same: a management API operation--any operation--should not return successfully until all observable side effects of that operation across a Pulsar cluster (including brokers, proxies, bookies, and ZK) were completed. All caches of metadata (e.g. on all brokers/proxies in the cluster) related to the operation should be cleared, and all persistent state (including ledger deletion, bookie cleanup, ZooKeeper metadata, etc.) should be updated *during* management API operations, and not afterwards.
If that means that management API operations take many seconds or minutes, that's still vastly preferable to not knowing when it is safe to interact with a cluster again after performing "DDL"-type changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] zbentley commented on issue #12551: Attempting to delete topics shortly after creation/deletion requests fails with a 500 error of "MetadataNotFoundException: Managed ledger not found"
Posted by GitBox <gi...@apache.org>.
zbentley commented on issue #12551:
URL: https://github.com/apache/pulsar/issues/12551#issuecomment-955748576
The following issues were all observed in response to similar testing:
https://github.com/apache/pulsar/issues/12557
https://github.com/apache/pulsar/issues/12556
https://github.com/apache/pulsar/issues/12555
https://github.com/apache/pulsar/issues/12554
https://github.com/apache/pulsar/issues/12553
https://github.com/apache/pulsar/issues/12552
https://github.com/apache/pulsar/issues/12551
The condition that caused these issues to occur appears to be interaction with various pulsar entities (e.g. creating/deleting things in the management API, or attempting to create consumers) *immediately after those entities were created* or *immediately after entities with the same name were deleted*.
I think the number of issues observed speaks to a defect in the management API functionality in general. Considering the severity of these issues (in many cases it is possible to force a topic/namespace into a permanently corrupted state), I hope a resolution can be found for the general/common root cause rather than fixing individual bug-inducing conditionls.
I suspect that the common root cause is that many management API operations are asynchronous that should not be.
Ideally, the resolution of all of these issues would be the same: a management API operation--any operation--should not return successfully until all observable side effects of that operation across a Pulsar cluster (including brokers, proxies, bookies, and ZK) were completed. All caches of metadata (e.g. on all brokers/proxies in the cluster) related to the operation should be cleared, and all persistent state (including ledger deletion, bookie cleanup, ZooKeeper metadata, etc.) should be updated *during* management API operations, and not afterwards.
If that means that management API operations take many seconds or minutes, that's still vastly preferable to not knowing when it is safe to interact with a cluster again after performing "DDL"-type changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] zbentley commented on issue #12551: Attempting to delete topics shortly after creation/deletion requests fails with a 500 error of "MetadataNotFoundException: Managed ledger not found"
Posted by GitBox <gi...@apache.org>.
zbentley commented on issue #12551:
URL: https://github.com/apache/pulsar/issues/12551#issuecomment-955749384
> > This is probably more important: all operations related to deletion/creation of topics/namespaces/tenants conclude before the management API returns success codes to the user.
>
> The operations are always acked after the writes are completed. What typically happens is that the 2nd request goes to a different broker which has not yet received a cache invalidation signal (eg: zk watch).
@merlimat that is good info. It indicates a partial workaround while these conditions are addressed; I can force clients doing "DDL" type work to always connect to a single broker.
However, that still will likely hit variations on these issues; since a given container entity (say a partitioned topic, or a namespace) can have sub-entities resident on multiple brokers, attempting to perform CRUD operations on entities on those other brokers (e.g. by having my client "chase" the 307 responses returned from the primary DDL broker when I attempt to do an action on something that's not mastered on that broker) will still encounter issues due to the other brokers' metadata caches not yet being updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org