You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/09/01 09:26:44 UTC
[GitHub] [pulsar] zbentley opened a new issue #11866: Attempting to delete a namespace immediately after deleting all topics in that namespace raises `org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty`
zbentley opened a new issue #11866:
URL: https://github.com/apache/pulsar/issues/11866
**Describe the bug**
If I delete a namespace (without `force`) via the V2 admin API immediately (less than a second) after issuing deletes for all topics in that namespace, I get a stacktrace and a 500 response code (full stacktrace at the bottom of this issue).
If I wait a moment (2 seconds is sufficient in all of my tests), that error no longer occurs when I attempt to delete the namespace.
It seems like there is an eventual-consistency condition that occurs after the final topic in a namespace is deleted, and that operation should instead be performed synchronously before topic-deletion returns a response.
**To Reproduce**
1. Ensure a broker is running. I used 2.8.0 standalone.
2. Create a namespace in the broker.
3. Create two or more topics in that namespace.
4. In parallel or in rapid succession, issue DELETE requests for both topics.
5. Immediately after recieving 2xx response codes from those topic DELETE requests, issue a DELETE request for the namespace itself.
6. Observe a 500 return code and stacktrace.
**Expected behavior**
The namespace should be successfully deleted.
Separately, all evidence of a topic's existence in the metadata store should be gone as of the end of a topic's DELETE handler.
**Desktop (please complete the following information):**
- OS: MacOS 10.11
An example stacktrace received from a namespace DELETE that occurred immediately after topic deletion:
```
E chariot.pulsar.client.exceptions.PulsarAPIError: 500 Server Error: Internal Server Error for url: http://localhost:8080/admin/v2/namespaces/chariot-test-obj-Site-4-PVRs-primary/chariot_ns_chariot-test-obj-ChariotTopic-5-mrFx
E Response code: 500
E URL: DELETE http://localhost:8080/admin/v2/namespaces/chariot-test-obj-Site-4-PVRs-primary/chariot_ns_chariot-test-obj-ChariotTopic-5-mrFx
E Headers: Headers({'host': 'localhost:8080', 'accept': '*/*', 'accept-encoding': 'gzip, deflate', 'connection': 'keep-alive', 'user-agent': 'python-httpx/0.18.2'})
E Payload: b''
E
E --- An unexpected error occurred in the server ---
E
E Message: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /namespace/chariot-test-obj-Site-4-PVRs-primary/chariot_ns_chariot-test-obj-ChariotTopic-5-mrFx
E
E Stacktrace:
E
E org.apache.pulsar.metadata.api.MetadataStoreException: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /namespace/chariot-test-obj-Site-4-PVRs-primary/chariot_ns_chariot-test-obj-ChariotTopic-5-mrFx
E at org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:345)
E at org.apache.pulsar.metadata.impl.ZKMetadataStore.lambda$storeDelete$22(ZKMetadataStore.java:308)
E at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
E at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
E at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
E at java.base/java.lang.Thread.run(Thread.java:834)
E Caused by: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /namespace/chariot-test-obj-Site-4-PVRs-primary/chariot_ns_chariot-test-obj-ChariotTopic-5-mrFx
E at org.apache.zookeeper.KeeperException.create(KeeperException.java:132)
E at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
E at org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:335)
E ... 5 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #11866: Attempting to delete a namespace immediately after deleting all topics in that namespace raises `org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty`
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on issue #11866:
URL: https://github.com/apache/pulsar/issues/11866#issuecomment-1058886661
The issue had no activity for 30 days, mark with Stale label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lordcheng10 removed a comment on issue #11866: Attempting to delete a namespace immediately after deleting all topics in that namespace raises `org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty`
Posted by GitBox <gi...@apache.org>.
lordcheng10 removed a comment on issue #11866:
URL: https://github.com/apache/pulsar/issues/11866#issuecomment-914577800
Can I take this up?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] zbentley commented on issue #11866: Attempting to delete a namespace immediately after deleting all topics in that namespace raises `org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty`
Posted by GitBox <gi...@apache.org>.
zbentley commented on issue #11866:
URL: https://github.com/apache/pulsar/issues/11866#issuecomment-909590977
This occurred against a standalone broker; are there multiple (e.g. thread local) metadata caches in use in standalone as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on issue #11866: Attempting to delete a namespace immediately after deleting all topics in that namespace raises `org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty`
Posted by GitBox <gi...@apache.org>.
lordcheng10 commented on issue #11866:
URL: https://github.com/apache/pulsar/issues/11866#issuecomment-914577800
Can I take this up?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] merlimat commented on issue #11866: Attempting to delete a namespace immediately after deleting all topics in that namespace raises `org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty`
Posted by GitBox <gi...@apache.org>.
merlimat commented on issue #11866:
URL: https://github.com/apache/pulsar/issues/11866#issuecomment-909558022
The reason is probably related to the caching of metadata in brokers and that the the different requests are hitting different brokers.
When you delete all topics, when you get the 200 responses, all the topics are gone. That will trigger notification to all brokers to update their cached views.
If the the delete namespace ends up in a different broker, its cache might still not have been updated.
There are some possible improvements there, but it can also be a bit dangerous to let some operations to bypass caches, as it could potentially lead to high load on Zookeeper.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] zbentley commented on issue #11866: Attempting to delete a namespace immediately after deleting all topics in that namespace raises `org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty`
Posted by GitBox <gi...@apache.org>.
zbentley commented on issue #11866:
URL: https://github.com/apache/pulsar/issues/11866#issuecomment-909590977
This occurred against a standalone broker; are there multiple (e.g. thread local) metadata caches in use in standalone as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] merlimat commented on issue #11866: Attempting to delete a namespace immediately after deleting all topics in that namespace raises `org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty`
Posted by GitBox <gi...@apache.org>.
merlimat commented on issue #11866:
URL: https://github.com/apache/pulsar/issues/11866#issuecomment-909558022
The reason is probably related to the caching of metadata in brokers and that the the different requests are hitting different brokers.
When you delete all topics, when you get the 200 responses, all the topics are gone. That will trigger notification to all brokers to update their cached views.
If the the delete namespace ends up in a different broker, its cache might still not have been updated.
There are some possible improvements there, but it can also be a bit dangerous to let some operations to bypass caches, as it could potentially lead to high load on Zookeeper.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org