You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/10/31 15:54:06 UTC

[GitHub] [pulsar] zbentley opened a new issue #12553: Attempting to delete a tenant that was recently created sometimes fails with "MetadataStoreException: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /admin/policies/$tenantname"

zbentley opened a new issue #12553:
URL: https://github.com/apache/pulsar/issues/12553


   **Describe the bug**
   When I create a tenant/namespace/topic/subscription and then immediately (seconds or less) try to delete them, I sometimes get a 500 response code and an internal stacktrace (see below).
   
   **To Reproduce**
   Run reproduction plan for https://github.com/apache/pulsar/issues/12551; sometimes the steps will not error, sometimes they will have the error described in that issue or others, and sometimes they will have this error.
   
   **Expected behavior**
   Tenant deletion should succeed if previous tenant creation succeeded, no matter how recently previous tenant creation was performed.
   
   **Environment**
   Same environment as https://github.com/apache/pulsar/issues/12551
   
   **Stacktrace**
   ```
   URL: DELETE http://pulsar-blt-sn-platform-proxy-headless.pulsar:8080/admin/v2/tenants/blt3
   Headers: Headers({'host': 'pulsar-blt-sn-platform-proxy-headless.pulsar:8080', 'accept': '*/*', 'accept-encoding': 'gzip, deflate', 'connection': 'keep-alive', 'user-agent': 'python-httpx/0.20.0'})
   Payload: b''
    --- An unexpected error occurred in the server ---
   
   Message: org.apache.pulsar.metadata.api.MetadataStoreException: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /admin/policies/blt3
   
   Stacktrace:
   
   java.util.concurrent.CompletionException: org.apache.pulsar.metadata.api.MetadataStoreException: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /admin/policies/blt3
   	at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331)
   	at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346)
   	at java.base/java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:777)
   	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
   	at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
   	at org.apache.pulsar.metadata.impl.ZKMetadataStore.lambda$null$21(ZKMetadataStore.java:277)
   	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
   	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
   	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
   	at java.base/java.lang.Thread.run(Thread.java:829)
   Caused by: org.apache.pulsar.metadata.api.MetadataStoreException: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /admin/policies/blt3
   	at org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:314)
   	... 5 more
   Caused by: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /admin/policies/blt3
   	at org.apache.zookeeper.KeeperException.create(KeeperException.java:132)
   	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
   	at org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:304)
   	... 5 more
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] github-actions[bot] commented on issue #12553: Attempting to delete a tenant that was recently created sometimes fails with "MetadataStoreException: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /admin/policies/$tenantname"

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #12553:
URL: https://github.com/apache/pulsar/issues/12553#issuecomment-1052938176


   The issue had no activity for 30 days, mark with Stale label.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] zbentley commented on issue #12553: Attempting to delete a tenant that was recently created sometimes fails with "MetadataStoreException: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /admin/policies/$tenantname"

Posted by GitBox <gi...@apache.org>.
zbentley commented on issue #12553:
URL: https://github.com/apache/pulsar/issues/12553#issuecomment-955748706


   The following issues were all observed in response to similar testing:
   https://github.com/apache/pulsar/issues/12557
   https://github.com/apache/pulsar/issues/12556
   https://github.com/apache/pulsar/issues/12555
   https://github.com/apache/pulsar/issues/12554
   https://github.com/apache/pulsar/issues/12553
   https://github.com/apache/pulsar/issues/12552
   https://github.com/apache/pulsar/issues/12551
   
   The condition that caused these issues to occur appears to be interaction with various pulsar entities (e.g. creating/deleting things in the management API, or attempting to create consumers) *immediately after those entities were created* or *immediately after entities with the same name were deleted*.
   
   I think the number of issues observed speaks to a defect in the management API functionality in general. Considering the severity of these issues (in many cases it is possible to force a topic/namespace into a permanently corrupted state), I hope a resolution can be found for the general/common root cause rather than fixing individual bug-inducing conditions.
   
   I suspect that the common root cause is that many management API operations are asynchronous that should not be.
   
   Ideally, the resolution of all of these issues would be the same: a management API operation--any operation--should not return successfully until all observable side effects of that operation across a Pulsar cluster (including brokers, proxies, bookies, and ZK) were completed. All caches of metadata (e.g. on all brokers/proxies in the cluster) related to the operation should be cleared, and all persistent state (including ledger deletion, bookie cleanup, ZooKeeper metadata, etc.) should be updated *during* management API operations, and not afterwards.
   
   If that means that management API operations take many seconds or minutes, that's still vastly preferable to not knowing when it is safe to interact with a cluster again after performing "DDL"-type changes.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] Technoboy- commented on issue #12553: Attempting to delete a tenant that was recently created sometimes fails with "MetadataStoreException: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /admin/policies/$tenantname"

Posted by GitBox <gi...@apache.org>.
Technoboy- commented on issue #12553:
URL: https://github.com/apache/pulsar/issues/12553#issuecomment-988544180


   The possible cause of this problem is that when tenants are being deleted, a new namespace is created, resulting in tenants having sub-nodes in the zk directory.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org