You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/04/19 09:19:38 UTC
[GitHub] [pulsar] frank-montyne opened a new issue #10263: Cannot delete non empty bundle
frank-montyne opened a new issue #10263:
URL: https://github.com/apache/pulsar/issues/10263
**Describe the bug**
Creating a tenant, namespace and topic and then deleting the topic, namespace and tenant again in quick succession using the Java client **_often_** fails at the moment of deletion of the namespace with the following exception:
org.apache.pulsar.client.admin.PulsarAdminException$ConflictException: Cannot delete non empty bundle
at org.apache.pulsar.client.admin.internal.BaseResource.getApiException(BaseResource.java:223)
at org.apache.pulsar.client.admin.internal.BaseResource$3.failed(BaseResource.java:187)
at org.apache.pulsar.shade.org.glassfish.jersey.client.JerseyInvocation$1.failed(JerseyInvocation.java:839)
at org.apache.pulsar.shade.org.glassfish.jersey.client.JerseyInvocation$1.completed(JerseyInvocation.java:820)
at org.apache.pulsar.shade.org.glassfish.jersey.client.ClientRuntime.processResponse(ClientRuntime.java:229)
at org.apache.pulsar.shade.org.glassfish.jersey.client.ClientRuntime.access$200(ClientRuntime.java:62)
at org.apache.pulsar.shade.org.glassfish.jersey.client.ClientRuntime$2.lambda$response$0(ClientRuntime.java:173)
at org.apache.pulsar.shade.org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
at org.apache.pulsar.shade.org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
at org.apache.pulsar.shade.org.glassfish.jersey.internal.Errors.process(Errors.java:292)
at org.apache.pulsar.shade.org.glassfish.jersey.internal.Errors.process(Errors.java:274)
at org.apache.pulsar.shade.org.glassfish.jersey.internal.Errors.process(Errors.java:244)
at org.apache.pulsar.shade.org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:288)
at org.apache.pulsar.shade.org.glassfish.jersey.client.ClientRuntime$2.response(ClientRuntime.java:173)
at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$apply$1(AsyncHttpConnector.java:212)
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2137)
at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$retryOperation$3(AsyncHttpConnector.java:253)
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2137)
at org.apache.pulsar.shade.org.asynchttpclient.netty.NettyResponseFuture.loadContent(NettyResponseFuture.java:222)
at org.apache.pulsar.shade.org.asynchttpclient.netty.NettyResponseFuture.done(NettyResponseFuture.java:257)
at org.apache.pulsar.shade.org.asynchttpclient.netty.handler.AsyncHttpClientHandler.finishUpdate(AsyncHttpClientHandler.java:241)
at org.apache.pulsar.shade.org.asynchttpclient.netty.handler.HttpHandler.handleChunk(HttpHandler.java:114)
at org.apache.pulsar.shade.org.asynchttpclient.netty.handler.HttpHandler.handleRead(HttpHandler.java:143)
at org.apache.pulsar.shade.org.asynchttpclient.netty.handler.AsyncHttpClientHandler.channelRead(AsyncHttpClientHandler.java:78)
at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at org.apache.pulsar.shade.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at org.apache.pulsar.shade.io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
at org.apache.pulsar.shade.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
at org.apache.pulsar.shade.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)
at org.apache.pulsar.shade.io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at org.apache.pulsar.shade.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at org.apache.pulsar.shade.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at org.apache.pulsar.shade.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at org.apache.pulsar.shade.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at org.apache.pulsar.shade.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at org.apache.pulsar.shade.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at org.apache.pulsar.shade.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at org.apache.pulsar.shade.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at org.apache.pulsar.shade.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at org.apache.pulsar.shade.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:832)
Caused by: org.apache.pulsar.shade.javax.ws.rs.ClientErrorException: HTTP 409 Conflict
at org.apache.pulsar.shade.org.glassfish.jersey.client.JerseyInvocation.createExceptionForFamily(JerseyInvocation.java:938)
at org.apache.pulsar.shade.org.glassfish.jersey.client.JerseyInvocation.convertToException(JerseyInvocation.java:921)
at org.apache.pulsar.shade.org.glassfish.jersey.client.JerseyInvocation.access$500(JerseyInvocation.java:77)
... 54 more
**To Reproduce**
Steps to reproduce the behavior:
Run the following code a few times to get the exception:
PulsarAdmin pulsarAdmin = PulsarAdmin.builder()
.serviceHttpUrl("http://localhost:8080")
.build();
List<String> allowedClusters = pulsarAdmin.clusters().getClusters();
TenantInfo tenantInfo = new TenantInfo(Set.of(), Set.copyOf(allowedClusters));
pulsarAdmin.tenants().createTenant("aTenant", tenantInfo);
pulsarAdmin.namespaces().createNamespace("aTenant/aNamespace");
pulsarAdmin.topics().createNonPartitionedTopic("persistent://aTenant/aNamespace/aTopic");
pulsarAdmin.topics().delete("persistent://aTenant/aNamespace/aTopic");
// Code below triggers exception.
**pulsarAdmin.namespaces().deleteNamespace("aTenant/aNamespace");**
pulsarAdmin.tenants().deleteTenant("aTenant");
**Expected behavior**
No exception
**Additional context**
Same behaviour when running Pulsar Docker 2.7.1 or Pulsar 2.7.1 locally from the command line on OSX.
I get the same error with non-persistent topics.
If I run in debug mode and wait some time before executing the line of code below I get no exception.
**pulsarAdmin.namespaces().deleteNamespace("aTenant/aNamespace");**
Added a part of the pulsar log file.
[log.txt](https://github.com/apache/pulsar/files/6334739/log.txt)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] MarvinCai commented on issue #10263: Cannot delete non empty bundle
Posted by GitBox <gi...@apache.org>.
MarvinCai commented on issue #10263:
URL: https://github.com/apache/pulsar/issues/10263#issuecomment-822913298
We're relying on Zookeeper watch to invalidate cache entry and refetch, so that delay is inevitable.
We might able to invalidate entry in cache before delete/update in Zookeeper, but currently metadata store is not aware of all the caches, so that'll be non trivial work and not sure how much value it brings.
Could use a second opinion here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] MarvinCai commented on issue #10263: Cannot delete non empty bundle
Posted by GitBox <gi...@apache.org>.
MarvinCai commented on issue #10263:
URL: https://github.com/apache/pulsar/issues/10263#issuecomment-823804453
I'm afraid `org.apache.pulsar.client.admin.PulsarAdminException$ConflictException: Cannot delete non empty bundle` is the only information we can get, pulsar can't tell that's because metadata cache not synced or the bundle is we actually have some more topics in the bundle.
I think the easiest work around might be just add a retry with some backoff.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Technoboy- commented on issue #10263: Cannot delete non empty bundle
Posted by GitBox <gi...@apache.org>.
Technoboy- commented on issue #10263:
URL: https://github.com/apache/pulsar/issues/10263#issuecomment-992088746
> There seems to be a connection internally in Pulsar(persistent://upDownDisk/disk/__change_events) to that prevents the delete from succeeding.
When I run the test locally on my MacBook Pro using the Pulsar docker image eventually the namespace is deleted after a number of retries. When I run the test on Jenkins the delte keeps failing for 90 seconds (=max wait time in the back-off strategy) and 91 tries (the delete is retried every second). I already tried to use longer wait times between retries it does not help.
yes, the root cause is that: Pulsar will create a system topic(__change_events) under every namespace. when you delete the namespace, this system topic may not close and block the delete operation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Technoboy- commented on issue #10263: Cannot delete non empty bundle
Posted by GitBox <gi...@apache.org>.
Technoboy- commented on issue #10263:
URL: https://github.com/apache/pulsar/issues/10263#issuecomment-992085870
Currently, you can set `forceDeleteNamespaceAllowed=true` in broker.conf and then try to delete the namespace forcibly.
We will optimize here later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] shibd commented on issue #10263: Cannot delete non empty bundle
Posted by GitBox <gi...@apache.org>.
shibd commented on issue #10263:
URL: https://github.com/apache/pulsar/issues/10263#issuecomment-992098097
> AFAIK I'm afraid org.apache.pulsar.client.admin.PulsarAdminException$ConflictException: Cannot delete non empty bundle is the only information we can get, pulsar can't tell that's because metadata cache not yet synced or we actually have some more topics in the bundle.
I think the easiest work around might be just add a retry with some backoff.
@MarvinCai When delete a namespace, can directly redirect the call `deletenamespacebundleforcefully`?
Because it has been checked that there is no user topic in the `deletenamespace`, do not need to check again when deleting namespacebundle.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] MarvinCai edited a comment on issue #10263: Cannot delete non empty bundle
Posted by GitBox <gi...@apache.org>.
MarvinCai edited a comment on issue #10263:
URL: https://github.com/apache/pulsar/issues/10263#issuecomment-823804453
AFAIK I'm afraid `org.apache.pulsar.client.admin.PulsarAdminException$ConflictException: Cannot delete non empty bundle` is the only information we can get, pulsar can't tell that's because metadata cache not yet synced or we actually have some more topics in the bundle.
I think the easiest work around might be just add a retry with some backoff.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] MarvinCai commented on issue #10263: Cannot delete non empty bundle
Posted by GitBox <gi...@apache.org>.
MarvinCai commented on issue #10263:
URL: https://github.com/apache/pulsar/issues/10263#issuecomment-822458610
Seems to be metadata syncing issue, the namespace service read metadata about topic from a cache, while the topic deletion operation finished and updated metadata in zookeeper, the change might not immediately reflected in the cache, hence you see the exception. Is it possible to add some wait time after the deletion? Or add a retry to the delete namespace operation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] MarvinCai edited a comment on issue #10263: Cannot delete non empty bundle
Posted by GitBox <gi...@apache.org>.
MarvinCai edited a comment on issue #10263:
URL: https://github.com/apache/pulsar/issues/10263#issuecomment-823804453
AFAIK I'm afraid `org.apache.pulsar.client.admin.PulsarAdminException$ConflictException: Cannot delete non empty bundle` is the only information we can get, pulsar can't tell that's because metadata cache not synced or the bundle is we actually have some more topics in the bundle.
I think the easiest work around might be just add a retry with some backoff.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] shibd edited a comment on issue #10263: Cannot delete non empty bundle
Posted by GitBox <gi...@apache.org>.
shibd edited a comment on issue #10263:
URL: https://github.com/apache/pulsar/issues/10263#issuecomment-992098097
> AFAIK I'm afraid org.apache.pulsar.client.admin.PulsarAdminException$ConflictException: Cannot delete non empty bundle is the only information we can get, pulsar can't tell that's because metadata cache not yet synced or we actually have some more topics in the bundle.
I think the easiest work around might be just add a retry with some backoff.
@MarvinCai When delete a namespace, can directly call `deletenamespacebundleforcefully`?
Because it has been checked that there is no user topic in the `deletenamespace`, do not need to check again when `deletenamespacebundle`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] frank-montyne edited a comment on issue #10263: Cannot delete non empty bundle
Posted by GitBox <gi...@apache.org>.
frank-montyne edited a comment on issue #10263:
URL: https://github.com/apache/pulsar/issues/10263#issuecomment-853728798
Example error:
I have a namespace with one topic. All Producers/consumers are closed. I delete the topic (no problem). Then I delete the namespace. When looking in the logs from Pulsar 2.7.2 I constantly get the following error while deleting the namespace:
10:41:34 PULSAR >08:41:34.550 [pulsar-30-2] ERROR org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - Deleting cursor
10:41:34 PULSAR >org.apache.bookkeeper.mledger.ManagedLedgerException$CursorNotFoundException: ManagedCursor not found: reader-8d468b1abe
10:41:34 PULSAR >08:41:34.550 [pulsar-30-2] WARN org.apache.pulsar.broker.service.persistent.PersistentSubscription - [persistent://upDownDisk/disk/__change_events] [reader-8d468b1abe] Failed to remove non durable cursor
10:41:34 PULSAR >org.apache.bookkeeper.mledger.ManagedLedgerException$CursorNotFoundException: ManagedCursor not found: reader-8d468b1abe
10:41:34 PULSAR >08:41:34.558 [AsyncHttpClient-104-1] WARN org.apache.pulsar.client.admin.internal.BaseResource - [http://localhost:7650/admin/v2/namespaces/upDownDisk/disk/0x00000000_0x40000000?force=false] Failed to perform http delete request: javax.ws.rs.ClientErrorException: HTTP 409 Conflict
There seems to be a connection internally in Pulsar(persistent://upDownDisk/disk/__change_events) to that prevents the delete from succeeding.
When I run the test locally on my MacBook Pro using the Pulsar docker image eventually the namespace is deleted after a number of retries. When I run the test on Jenkins the delte keeps failing for 90 seconds (=max wait time in the back-off strategy) and 91 tries (the delete is retried every second). I already tried to use longer wait times between retries it does not help.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] frank-montyne edited a comment on issue #10263: Cannot delete non empty bundle
Posted by GitBox <gi...@apache.org>.
frank-montyne edited a comment on issue #10263:
URL: https://github.com/apache/pulsar/issues/10263#issuecomment-853728798
Example error:
I have a namespace with one topic. All Producers/consumers are closed. I delete the topic (no problem). Then I delete the namespace. When looking in the logs from Pulsar 2.7.2 I constantly get the following error while deleting the namespace:
10:41:34 PULSAR >08:41:34.550 [pulsar-30-2] ERROR org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - Deleting cursor
10:41:34 PULSAR >org.apache.bookkeeper.mledger.ManagedLedgerException$CursorNotFoundException: ManagedCursor not found: reader-8d468b1abe
10:41:34 PULSAR >08:41:34.550 [pulsar-30-2] WARN org.apache.pulsar.broker.service.persistent.PersistentSubscription - [persistent://upDownDisk/disk/__change_events] [reader-8d468b1abe] Failed to remove non durable cursor
10:41:34 PULSAR >org.apache.bookkeeper.mledger.ManagedLedgerException$CursorNotFoundException: ManagedCursor not found: reader-8d468b1abe
10:41:34 PULSAR >08:41:34.558 [AsyncHttpClient-104-1] WARN org.apache.pulsar.client.admin.internal.BaseResource - [http://localhost:7650/admin/v2/namespaces/upDownDisk/disk/0x00000000_0x40000000?force=false] Failed to perform http delete request: javax.ws.rs.ClientErrorException: HTTP 409 Conflict
There seems to be a connection internally in Pulsar(persistent://upDownDisk/disk/__change_events) to that prevents the delete from succeeding.
When I run the test locally on my MacBook Pro using the Pulsar docker image eventually the namespace is deleted after a number of retries. When I run the test on Jenkins the delte keeps failing for 90 seconds (=max wait time in the back-off strategy) and 91 tries (the delete is retried every second). I already tried to use longer wait times between retries it does not help.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] MarvinCai edited a comment on issue #10263: Cannot delete non empty bundle
Posted by GitBox <gi...@apache.org>.
MarvinCai edited a comment on issue #10263:
URL: https://github.com/apache/pulsar/issues/10263#issuecomment-823804453
AFAIK I'm afraid `org.apache.pulsar.client.admin.PulsarAdminException$ConflictException: Cannot delete non empty bundle` is the only information we can get, pulsar can't tell that's because metadata cache not synced or we actually have some more topics in the bundle.
I think the easiest work around might be just add a retry with some backoff.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] shibd edited a comment on issue #10263: Cannot delete non empty bundle
Posted by GitBox <gi...@apache.org>.
shibd edited a comment on issue #10263:
URL: https://github.com/apache/pulsar/issues/10263#issuecomment-992098097
> AFAIK I'm afraid org.apache.pulsar.client.admin.PulsarAdminException$ConflictException: Cannot delete non empty bundle is the only information we can get, pulsar can't tell that's because metadata cache not yet synced or we actually have some more topics in the bundle.
I think the easiest work around might be just add a retry with some backoff.
@MarvinCai When delete a namespace, can directly redirect the call `deletenamespacebundleforcefully`?
Because it has been checked that there is no user topic in the `deletenamespace`, do not need to check again when `deletenamespacebundle`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] shibd edited a comment on issue #10263: Cannot delete non empty bundle
Posted by GitBox <gi...@apache.org>.
shibd edited a comment on issue #10263:
URL: https://github.com/apache/pulsar/issues/10263#issuecomment-992098097
> AFAIK I'm afraid org.apache.pulsar.client.admin.PulsarAdminException$ConflictException: Cannot delete non empty bundle is the only information we can get, pulsar can't tell that's because metadata cache not yet synced or we actually have some more topics in the bundle.
I think the easiest work around might be just add a retry with some backoff.
@MarvinCai When delete a namespace, can directly redirect `deletenamespacebundleforcefully` method?
Because it has been checked that there is no user topic in the `deletenamespace`, do not need to check again when `deletenamespacebundle`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] frank-montyne commented on issue #10263: Cannot delete non empty bundle
Posted by GitBox <gi...@apache.org>.
frank-montyne commented on issue #10263:
URL: https://github.com/apache/pulsar/issues/10263#issuecomment-853675538
@MarvinCai is there any way to force a metadata cache update or to increase the refresh rate? This remains a problem in our test environment that is very dynamic in nature and where tenants, namspaces and topics get created and deleted all the time. The fact that there is no indication of a real error versus a cache sync issue keeps giving us problems. Depending on the environment in which the tests run it can take 30 seconds to 5 minutes before the cache is in sync again causing the tests in one case to fail and in another to succeed because in the back-off strategy we maximally wait one minute. We don't want to wait longer than that because otherwise in case of a real error it would take an unacceptable amount of time before the delete action fails. Any help or workaround is appreciated!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] frank-montyne commented on issue #10263: Cannot delete non empty bundle
Posted by GitBox <gi...@apache.org>.
frank-montyne commented on issue #10263:
URL: https://github.com/apache/pulsar/issues/10263#issuecomment-822747814
I can do that but that’s quite arbitrary. Is it possible to fix this in the future?
> On 19 Apr 2021, at 15:16, Marvin Cai ***@***.***> wrote:
>
>
> Seems to be metadata syncing issue, the namespace service read metadata about topic from a cache, while the topic deletion operation finished and updated metadata in zookeeper, the change might not immediately reflected in the cache, hence you see the exception. Is it possible to add some wait time after the deletion? Or add a retry to the delete namespace operation.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub, or unsubscribe.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] frank-montyne commented on issue #10263: Cannot delete non empty bundle
Posted by GitBox <gi...@apache.org>.
frank-montyne commented on issue #10263:
URL: https://github.com/apache/pulsar/issues/10263#issuecomment-823210500
If I add a retry is there a way to know that I got the exception due to the
fact that the cache was not up to date yet? If not how can I discriminate
between a real error and a cache problem?
On Tue, Apr 20, 2021 at 3:58 AM Marvin Cai ***@***.***> wrote:
> We're relying on Zookeeper watch to invalidate cache entry and refetch, so
> that delay is inevitable.
> We might able to invalidate entry in cache before delete/update in
> Zookeeper, but currently metadata store is not aware of all the caches, so
> that'll be non trivial work and not sure how much value it brings.
> Could use a second opinion here.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/pulsar/issues/10263#issuecomment-822913298>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ACJKQL5JCJIOAVFOMWJUGQ3TJTNS3ANCNFSM43FNPO6A>
> .
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] frank-montyne commented on issue #10263: Cannot delete non empty bundle
Posted by GitBox <gi...@apache.org>.
frank-montyne commented on issue #10263:
URL: https://github.com/apache/pulsar/issues/10263#issuecomment-853675538
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] frank-montyne commented on issue #10263: Cannot delete non empty bundle
Posted by GitBox <gi...@apache.org>.
frank-montyne commented on issue #10263:
URL: https://github.com/apache/pulsar/issues/10263#issuecomment-853728798
Example error:
I have a namespace with one topic. All Producers/consumers are cloed. I delete the topic (no problem). Then I delete the namespace. When looking in the logs from Pulsar 2.7.2 I constantly get the following error while deleting the namespace:
10:41:34 PULSAR >08:41:34.550 [pulsar-30-2] ERROR org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - Deleting cursor
10:41:34 PULSAR >org.apache.bookkeeper.mledger.ManagedLedgerException$CursorNotFoundException: ManagedCursor not found: reader-8d468b1abe
10:41:34 PULSAR >08:41:34.550 [pulsar-30-2] WARN org.apache.pulsar.broker.service.persistent.PersistentSubscription - [persistent://upDownDisk/disk/__change_events] [reader-8d468b1abe] Failed to remove non durable cursor
10:41:34 PULSAR >org.apache.bookkeeper.mledger.ManagedLedgerException$CursorNotFoundException: ManagedCursor not found: reader-8d468b1abe
10:41:34 PULSAR >08:41:34.558 [AsyncHttpClient-104-1] WARN org.apache.pulsar.client.admin.internal.BaseResource - [http://localhost:7650/admin/v2/namespaces/upDownDisk/disk/0x00000000_0x40000000?force=false] Failed to perform http delete request: javax.ws.rs.ClientErrorException: HTTP 409 Conflict
There seems to be a connection internally in Pulsar(persistent://upDownDisk/disk/__change_events) to that prevents the delete from succeeding.
When I run the test locally on my MacBook Pro using the Pulsar docker image eventually the namespace is deleted after a number of retries. When I run the test on Jenkins the delte keeps failing for 90 seconds (=max wait time in the back-off strategy) and 91 tries (the delete is retried every second). I already tried to use longer wait times between retries it does not help.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #10263: Cannot delete non empty bundle
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on issue #10263:
URL: https://github.com/apache/pulsar/issues/10263#issuecomment-1058891461
The issue had no activity for 30 days, mark with Stale label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Technoboy- closed issue #10263: Cannot delete non empty bundle
Posted by GitBox <gi...@apache.org>.
Technoboy- closed issue #10263:
URL: https://github.com/apache/pulsar/issues/10263
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org