You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "lhotari (via GitHub)" <gi...@apache.org> on 2023/07/07 19:41:56 UTC
[GitHub] [pulsar] lhotari opened a new issue, #20376: Flaky-test: AdminApi2Test.resetClusters
lhotari opened a new issue, #20376:
URL: https://github.com/apache/pulsar/issues/20376
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Example failure
https://github.com/apache/pulsar/actions/runs/5056774399/jobs/9075132818?pr=20372#step:10:1446
### Exception stacktrace
<!-- copy-paste the stack trace in the code block below -->
```
Error: Tests run: 82, Failures: 1, Errors: 0, Skipped: 55, Time elapsed: 660.065 s <<< FAILURE! - in org.apache.pulsar.broker.admin.AdminApi2Test
Error: org.apache.pulsar.broker.admin.AdminApi2Test.resetClusters Time elapsed: 2.209 s <<< FAILURE!
org.apache.pulsar.client.admin.PulsarAdminException$ServerSideErrorException:
--- An unexpected error occurred in the server ---
Message: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /managed-ledgers/prop-xyz2
Stacktrace:
org.apache.pulsar.metadata.api.MetadataStoreException: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /managed-ledgers/prop-xyz2
at org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:476)
at org.apache.pulsar.metadata.impl.ZKMetadataStore.handleDeleteResult(ZKMetadataStore.java:304)
at org.apache.pulsar.metadata.impl.ZKMetadataStore.lambda$batchOperation$5(ZKMetadataStore.java:216)
at org.apache.zookeeper.MockZooKeeper.multi(MockZooKeeper.java:1006)
at org.apache.zookeeper.MockZooKeeperSession.multi(MockZooKeeperSession.java:191)
at org.apache.pulsar.metadata.impl.ZKMetadataStore.batchOperation(ZKMetadataStore.java:190)
at org.apache.pulsar.metadata.impl.batching.AbstractBatchedMetadataStore.internalBatchOperation(AbstractBatchedMetadataStore.java:184)
at org.apache.pulsar.metadata.impl.batching.AbstractBatchedMetadataStore.flush(AbstractBatchedMetadataStore.java:124)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /managed-ledgers/prop-xyz2
at org.apache.zookeeper.KeeperException.create(KeeperException.java:132)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
at org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:466)
... 14 more
at org.apache.pulsar.client.admin.PulsarAdminException.wrap(PulsarAdminException.java:252)
at org.apache.pulsar.client.admin.internal.BaseResource.sync(BaseResource.java:352)
at org.apache.pulsar.client.admin.internal.TenantsImpl.deleteTenant(TenantsImpl.java:93)
at org.apache.pulsar.broker.admin.AdminApi2Test.resetClusters(AdminApi2Test.java:181)
at jdk.internal.reflect.GeneratedMethodAccessor548.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.testng.internal.invokers.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:139)
at org.testng.internal.invokers.MethodInvocationHelper.invokeMethodConsideringTimeout(MethodInvocationHelper.java:69)
at org.testng.internal.invokers.ConfigInvoker.invokeConfigurationMethod(ConfigInvoker.java:361)
at org.testng.internal.invokers.ConfigInvoker.invokeConfigurations(ConfigInvoker.java:296)
at org.testng.internal.invokers.TestInvoker.runConfigMethods(TestInvoker.java:823)
at org.testng.internal.invokers.TestInvoker.runAfterConfigurations(TestInvoker.java:792)
at org.testng.internal.invokers.TestInvoker.invokeMethod(TestInvoker.java:768)
at org.testng.internal.invokers.TestInvoker.invokeTestMethod(TestInvoker.java:221)
at org.testng.internal.invokers.MethodRunner.runInSequence(MethodRunner.java:50)
at org.testng.internal.invokers.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:969)
at org.testng.internal.invokers.TestInvoker.invokeTestMethods(TestInvoker.java:194)
at org.testng.internal.invokers.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:148)
at org.testng.internal.invokers.TestMethodWorker.run(TestMethodWorker.java:128)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at org.testng.TestRunner.privateRun(TestRunner.java:829)
at org.testng.TestRunner.run(TestRunner.java:602)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:437)
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:431)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:391)
at org.testng.SuiteRunner.run(SuiteRunner.java:330)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:95)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1256)
at org.testng.TestNG.runSuitesLocally(TestNG.java:1176)
at org.testng.TestNG.runSuites(TestNG.java:1099)
at org.testng.TestNG.run(TestNG.java:1067)
at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:155)
at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeSingleClass(TestNGDirectoryTestSuite.java:102)
at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeLazy(TestNGDirectoryTestSuite.java:117)
at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:86)
at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:137)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:385)
at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:507)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:495)
Suppressed: org.apache.pulsar.client.admin.PulsarAdminException$ServerSideErrorException:
--- An unexpected error occurred in the server ---
Message: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /managed-ledgers/prop-xyz2
Stacktrace:
org.apache.pulsar.metadata.api.MetadataStoreException: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /managed-ledgers/prop-xyz2
at org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:476)
at org.apache.pulsar.metadata.impl.ZKMetadataStore.handleDeleteResult(ZKMetadataStore.java:304)
at org.apache.pulsar.metadata.impl.ZKMetadataStore.lambda$batchOperation$5(ZKMetadataStore.java:216)
at org.apache.zookeeper.MockZooKeeper.multi(MockZooKeeper.java:1006)
at org.apache.zookeeper.MockZooKeeperSession.multi(MockZooKeeperSession.java:191)
at org.apache.pulsar.metadata.impl.ZKMetadataStore.batchOperation(ZKMetadataStore.java:190)
at org.apache.pulsar.metadata.impl.batching.AbstractBatchedMetadataStore.internalBatchOperation(AbstractBatchedMetadataStore.java:184)
at org.apache.pulsar.metadata.impl.batching.AbstractBatchedMetadataStore.flush(AbstractBatchedMetadataStore.java:124)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /managed-ledgers/prop-xyz2
at org.apache.zookeeper.KeeperException.create(KeeperException.java:132)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
at org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:466)
... 14 more
at org.apache.pulsar.client.admin.internal.BaseResource.getApiException(BaseResource.java:272)
at org.apache.pulsar.client.admin.internal.BaseResource$4.failed(BaseResource.java:237)
at org.glassfish.jersey.client.JerseyInvocation$1.failed(JerseyInvocation.java:882)
at org.glassfish.jersey.client.JerseyInvocation$1.completed(JerseyInvocation.java:863)
at org.glassfish.jersey.client.ClientRuntime.processResponse(ClientRuntime.java:229)
at org.glassfish.jersey.client.ClientRuntime.access$200(ClientRuntime.java:62)
at org.glassfish.jersey.client.ClientRuntime$2.lambda$response$0(ClientRuntime.java:173)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:288)
at org.glassfish.jersey.client.ClientRuntime$2.response(ClientRuntime.java:173)
at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$apply$1(AsyncHttpConnector.java:254)
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$retryOperation$4(AsyncHttpConnector.java:296)
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
at org.asynchttpclient.netty.NettyResponseFuture.loadContent(NettyResponseFuture.java:222)
at org.asynchttpclient.netty.NettyResponseFuture.done(NettyResponseFuture.java:257)
at org.asynchttpclient.netty.handler.AsyncHttpClientHandler.finishUpdate(AsyncHttpClientHandler.java:241)
at org.asynchttpclient.netty.handler.HttpHandler.handleChunk(HttpHandler.java:114)
at org.asynchttpclient.netty.handler.HttpHandler.handleRead(HttpHandler.java:143)
at org.asynchttpclient.netty.handler.AsyncHttpClientHandler.channelRead(AsyncHttpClientHandler.java:78)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: javax.ws.rs.InternalServerErrorException: HTTP 500 {"reason":"\n --- An unexpected error occurred in the server ---\n\nMessage: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /managed-ledgers/prop-xyz2\n\nStacktrace:\n\norg.apache.pulsar.metadata.api.MetadataStoreException: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /managed-ledgers/prop-xyz2\n\tat org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:476)\n\tat org.apache.pulsar.metadata.impl.ZKMetadataStore.handleDeleteResult(ZKMetadataStore.java:304)\n\tat org.apache.pulsar.metadata.impl.ZKMetadataStore.lambda$batchOperation$5(ZKMetadataStore.java:216)\n\tat org.apache.zookeeper.MockZooKeeper.multi(MockZooKeeper.java:1006)\n\tat org.apache.zookeeper.MockZooKeeperSession.multi(MockZooKeeperSession.java:191)\n\tat org.apache.pulsar.metadata.impl.ZKMetadataStore.batchOperation(ZKMetada
taStore.java:190)\n\tat org.apache.pulsar.metadata.impl.batching.AbstractBatchedMetadataStore.internalBatchOperation(AbstractBatchedMetadataStore.java:184)\n\tat org.apache.pulsar.metadata.impl.batching.AbstractBatchedMetadataStore.flush(AbstractBatchedMetadataStore.java:124)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)\n\tat java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)\n\tat java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\nCaused by: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode =
Directory not empty for /managed-ledgers/prop-xyz2\n\tat org.apache.zookeeper.KeeperException.create(KeeperException.java:132)\n\tat org.apache.zookeeper.KeeperException.create(KeeperException.java:54)\n\tat org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:466)\n\t... 14 more\n"}
at org.glassfish.jersey.client.JerseyInvocation.convertToException(JerseyInvocation.java:960)
at org.glassfish.jersey.client.JerseyInvocation.access$700(JerseyInvocation.java:82)
... 54 more
Caused by: [CIRCULAR REFERENCE: javax.ws.rs.InternalServerErrorException: HTTP 500 {"reason":"\n --- An unexpected error occurred in the server ---\n\nMessage: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /managed-ledgers/prop-xyz2\n\nStacktrace:\n\norg.apache.pulsar.metadata.api.MetadataStoreException: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /managed-ledgers/prop-xyz2\n\tat org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:476)\n\tat org.apache.pulsar.metadata.impl.ZKMetadataStore.handleDeleteResult(ZKMetadataStore.java:304)\n\tat org.apache.pulsar.metadata.impl.ZKMetadataStore.lambda$batchOperation$5(ZKMetadataStore.java:216)\n\tat org.apache.zookeeper.MockZooKeeper.multi(MockZooKeeper.java:1006)\n\tat org.apache.zookeeper.MockZooKeeperSession.multi(MockZooKeeperSession.java:191)\n\tat org.apache.pulsar.metadata.impl.ZKMetadataStore.bat
chOperation(ZKMetadataStore.java:190)\n\tat org.apache.pulsar.metadata.impl.batching.AbstractBatchedMetadataStore.internalBatchOperation(AbstractBatchedMetadataStore.java:184)\n\tat org.apache.pulsar.metadata.impl.batching.AbstractBatchedMetadataStore.flush(AbstractBatchedMetadataStore.java:124)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)\n\tat java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)\n\tat java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\nCaused by: org.apache.zookeeper.KeeperException$NotEmptyException
: KeeperErrorCode = Directory not empty for /managed-ledgers/prop-xyz2\n\tat org.apache.zookeeper.KeeperException.create(KeeperException.java:132)\n\tat org.apache.zookeeper.KeeperException.create(KeeperException.java:54)\n\tat org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:466)\n\t... 14 more\n"}]
```
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michaeljmarshall commented on issue #20376: Flaky-test: AdminApi2Test.resetClusters
Posted by "michaeljmarshall (via GitHub)" <gi...@apache.org>.
michaeljmarshall commented on issue #20376:
URL: https://github.com/apache/pulsar/issues/20376#issuecomment-1562021204
#20392 might help. Although, it looks like changing how a partitioned topic is classified only results in the topic being deleted _later_ in the process. Perhaps we need to look at how the topic policies client is updated? It seems like the client should be notified that the namespace is getting deleted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michaeljmarshall commented on issue #20376: Flaky-test: AdminApi2Test.resetClusters
Posted by "michaeljmarshall (via GitHub)" <gi...@apache.org>.
michaeljmarshall commented on issue #20376:
URL: https://github.com/apache/pulsar/issues/20376#issuecomment-1562265524
After further analysis, I think that #20392 is very likely to fix this flaky test, or at least one cause of this flaky test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michaeljmarshall commented on issue #20376: Flaky-test: AdminApi2Test.resetClusters
Posted by "michaeljmarshall (via GitHub)" <gi...@apache.org>.
michaeljmarshall commented on issue #20376:
URL: https://github.com/apache/pulsar/issues/20376#issuecomment-1561535497
Interestingly, there are mailing list threads about this subject from a while ago: https://lists.apache.org/thread/glc0c4dbcpv2sfccmdz5hm5tdqk4ms75
That thread makes me wonder if the tenant deletion is issuing multiple requests to delete child nodes and then issuing requests to delete parent nodes without waiting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari closed issue #20376: Flaky-test: AdminApi2Test.resetClusters
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari closed issue #20376: Flaky-test: AdminApi2Test.resetClusters
URL: https://github.com/apache/pulsar/issues/20376
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michaeljmarshall commented on issue #20376: Flaky-test: AdminApi2Test.resetClusters
Posted by "michaeljmarshall (via GitHub)" <gi...@apache.org>.
michaeljmarshall commented on issue #20376:
URL: https://github.com/apache/pulsar/issues/20376#issuecomment-1561521626
The stack trace is similar to https://github.com/apache/pulsar/issues/12553, which hints at issues with the metadata model being eventually consistent (which would explain why retries on the namespace deletion resolved the flakiness in the test).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on issue #20376: Flaky-test: AdminApi2Test.resetClusters
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on issue #20376:
URL: https://github.com/apache/pulsar/issues/20376#issuecomment-1577419278
> I haven't been able to reproduce this locally. It would be really valuable to know the zk tree structure at this time of failure. I wonder if there is an easy way to get that so we can debug with more information. Perhaps we can enable zk event logs?
Perhaps running the test in docker with limited resources would reproduce it. More details: https://github.com/apache/pulsar/issues/10380#issuecomment-826849810
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Flaky-test: AdminApi2Test.resetClusters [pulsar]
Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat commented on issue #20376:
URL: https://github.com/apache/pulsar/issues/20376#issuecomment-6376822820
Thanks @lhotari for reporting this. #20529 fixed it (milestone 3.1.0): resetClusters now restarts the test broker when tenant/namespace cleanup fails. #20392 fixed the classification of partitioned `__change_events` topics that tenant deletion depends on. If AdminApi2Test.resetClusters fails again on master, please reopen or open a new issue with the failure link.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michaeljmarshall commented on issue #20376: Flaky-test: AdminApi2Test.resetClusters
Posted by "michaeljmarshall (via GitHub)" <gi...@apache.org>.
michaeljmarshall commented on issue #20376:
URL: https://github.com/apache/pulsar/issues/20376#issuecomment-1560427090
Another: https://github.com/apache/pulsar/actions/runs/5059985368/jobs/9085290991?pr=20377#step:10:1656
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michaeljmarshall commented on issue #20376: Flaky-test: AdminApi2Test.resetClusters
Posted by "michaeljmarshall (via GitHub)" <gi...@apache.org>.
michaeljmarshall commented on issue #20376:
URL: https://github.com/apache/pulsar/issues/20376#issuecomment-1561511023
Looks like the namespace deletion was also an issue, and that was "solved" by adding retry logic in the aforementioned code. See https://github.com/apache/pulsar/pull/18220.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michaeljmarshall commented on issue #20376: Flaky-test: AdminApi2Test.resetClusters
Posted by "michaeljmarshall (via GitHub)" <gi...@apache.org>.
michaeljmarshall commented on issue #20376:
URL: https://github.com/apache/pulsar/issues/20376#issuecomment-1571063489
Another https://github.com/apache/pulsar/actions/runs/5092810099/jobs/9245740664?pr=20404#step:10:1496
```
Error: Tests run: 105, Failures: 1, Errors: 0, Skipped: 25, Time elapsed: 575.929 s <<< FAILURE! - in org.apache.pulsar.client.api.SimpleProducerConsumerTest
Error: org.apache.pulsar.client.api.SimpleProducerConsumerTest.rest Time elapsed: 85.692 s <<< FAILURE!
org.testcontainers.shaded.org.awaitility.core.ConditionTimeoutException: Condition with org.apache.pulsar.broker.auth.MockedPulsarServiceBaseTest was not fulfilled within 5 seconds.
at org.testcontainers.shaded.org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:167)
at org.testcontainers.shaded.org.awaitility.core.CallableCondition.await(CallableCondition.java:78)
at org.testcontainers.shaded.org.awaitility.core.CallableCondition.await(CallableCondition.java:26)
at org.testcontainers.shaded.org.awaitility.core.ConditionFactory.until(ConditionFactory.java:985)
at org.testcontainers.shaded.org.awaitility.core.ConditionFactory.until(ConditionFactory.java:954)
at org.apache.pulsar.broker.auth.MockedPulsarServiceBaseTest.deleteNamespaceWithRetry(MockedPulsarServiceBaseTest.java:618)
at org.apache.pulsar.broker.auth.MockedPulsarServiceBaseTest.deleteNamespaceWithRetry(MockedPulsarServiceBaseTest.java:609)
at org.apache.pulsar.broker.auth.MockedPulsarServiceBaseTest.deleteNamespaceWithRetry(MockedPulsarServiceBaseTest.java:593)
at org.apache.pulsar.client.api.SimpleProducerConsumerTest.rest(SimpleProducerConsumerTest.java:147)
at jdk.internal.reflect.GeneratedMethodAccessor414.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.testng.internal.invokers.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:139)
at org.testng.internal.invokers.MethodInvocationHelper.invokeMethodConsideringTimeout(MethodInvocationHelper.java:69)
at org.testng.internal.invokers.ConfigInvoker.invokeConfigurationMethod(ConfigInvoker.java:361)
at org.testng.internal.invokers.ConfigInvoker.invokeConfigurations(ConfigInvoker.java:296)
at org.testng.internal.invokers.TestInvoker.runConfigMethods(TestInvoker.java:823)
at org.testng.internal.invokers.TestInvoker.runAfterConfigurations(TestInvoker.java:792)
at org.testng.internal.invokers.TestInvoker.invokeMethod(TestInvoker.java:768)
at org.testng.internal.invokers.TestInvoker.invokeTestMethod(TestInvoker.java:221)
at org.testng.internal.invokers.MethodRunner.runInSequence(MethodRunner.java:50)
at org.testng.internal.invokers.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:969)
at org.testng.internal.invokers.TestInvoker.invokeTestMethods(TestInvoker.java:194)
at org.testng.internal.invokers.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:148)
at org.testng.internal.invokers.TestMethodWorker.run(TestMethodWorker.java:128)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at org.testng.TestRunner.privateRun(TestRunner.java:829)
at org.testng.TestRunner.run(TestRunner.java:602)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:437)
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:431)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:391)
at org.testng.SuiteRunner.run(SuiteRunner.java:330)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:95)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1256)
at org.testng.TestNG.runSuitesLocally(TestNG.java:1176)
at org.testng.TestNG.runSuites(TestNG.java:1099)
at org.testng.TestNG.run(TestNG.java:1067)
at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:155)
at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeSingleClass(TestNGDirectoryTestSuite.java:102)
at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeLazy(TestNGDirectoryTestSuite.java:117)
at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:86)
at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:137)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:385)
at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:507)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:495)
Caused by: java.util.concurrent.TimeoutException
at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:204)
at org.testcontainers.shaded.org.awaitility.core.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:101)
at org.testcontainers.shaded.org.awaitility.core.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:81)
at org.testcontainers.shaded.org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:103)
... 46 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on issue #20376: Flaky-test: AdminApi2Test.resetClusters
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #20376:
URL: https://github.com/apache/pulsar/issues/20376#issuecomment-1667069911
The issue had no activity for 30 days, mark with Stale label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michaeljmarshall commented on issue #20376: Flaky-test: AdminApi2Test.resetClusters
Posted by "michaeljmarshall (via GitHub)" <gi...@apache.org>.
michaeljmarshall commented on issue #20376:
URL: https://github.com/apache/pulsar/issues/20376#issuecomment-1561585239
Here is the rest of my error log file:
[org.apache.pulsar.broker.admin.AdminApi2Test-output.txt](https://github.com/apache/pulsar/files/11557017/org.apache.pulsar.broker.admin.AdminApi2Test-output.txt)
With additional inspection, it seems like part of the problem could easily be system topics. I see issues related to `__change-events` topic:
```
xyz2/ns1/persistent/__change_events-partition-11, ackPos=66:-1, readPos=66:0}]
2023-05-24T15:30:44,862 - INFO - [PulsarTestContext-executor-OrderedExecutor-0-0:PersistentSubscription@312] - [persistent://prop-xyz2/ns1/__change_events-partition-11][multiTopicsReader-1b06c712cc] Successfully closed dispatcher for reader
2023-05-24T15:30:44,862 - ERROR - [PulsarTestContext-executor-OrderedExecutor-0-0:PersistentTopic@948] - [persistent://prop-xyz2/ns1/__change_events-partition-11] Failed to create subscription: multiTopicsReader-1b06c712cc
java.util.concurrent.CompletionException: org.apache.pulsar.broker.service.BrokerServiceException: Connection was closed while the opening the cursor
at java.util.concurrent.CompletableFuture.encodeRelay(CompletableFuture.java:368) ~[?:?]
at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1189) ~[?:?]
at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2309) ~[?:?]
at org.apache.pulsar.broker.service.persistent.PersistentTopic.lambda$internalSubscribe$20(PersistentTopic.java:890) ~[classes/:?]
at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1187) ~[?:?]
at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2309) ~[?:?]
at org.apache.pulsar.broker.service.persistent.PersistentTopic.lambda$internalSubscribe$22(PersistentTopic.java:885) ~[classes/:?]
at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1187) ~[?:?]
at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2309) ~[?:?]
at org.apache.pulsar.broker.service.persistent.PersistentTopic.internalSubscribe(PersistentTopic.java:806) ~[classes/:?]
at org.apache.pulsar.broker.service.persistent.PersistentTopic.subscribe(PersistentTopic.java:780) ~[classes/:?]
at org.apache.pulsar.broker.service.ServerCnx.lambda$handleSubscribe$14(ServerCnx.java:1239) ~[classes/:?]
at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147) ~[?:?]
at org.apache.pulsar.broker.service.schema.BookkeeperSchemaStorage.lambda$closeLedger$42(BookkeeperSchemaStorage.java:619) ~[classes/:?]
at org.apache.bookkeeper.client.PulsarMockLedgerHandle.lambda$asyncClose$1(PulsarMockLedgerHandle.java:96) ~[testmocks-3.1.0-SNAPSHOT.jar:4.16.1]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?]
at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482) ~[?:?]
at org.apache.bookkeeper.common.util.SingleThreadExecutor.safeRunTask(SingleThreadExecutor.java:137) ~[bookkeeper-common-4.16.1.jar:4.16.1]
at org.apache.bookkeeper.common.util.SingleThreadExecutor.run(SingleThreadExecutor.java:107) ~[bookkeeper-common-4.16.1.jar:4.16.1]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.89.Final.jar:4.1.89.Final]
at java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: org.apache.pulsar.broker.service.BrokerServiceException: Connection was closed while the opening the cursor
at org.apache.pulsar.broker.service.persistent.PersistentTopic.lambda$internalSubscribe$19(PersistentTopic.java:914) ~[classes/:?]
at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1187) ~[?:?]
... 22 more
2023-05-24T15:30:44,864 - WARN - [PulsarTestContext-executor-OrderedExecutor-0-0:ServerCnx@1282] - [/127.0.0.1:57364][persistent://prop-xyz2/ns1/__change_events-partition-11][multiTopicsReader-1b06c712cc] Failed to create consumer: consumerId=11, Connection was closed while the opening the cursor
2023-05-24T15:30:44,864 - INFO - [PulsarTestContext-executor-OrderedExecutor-0-0:PersistentTopic@1033] - [persistent://prop-xyz2/ns1/__change_events-partition-4][multiTopicsReader-1b06c712cc] Creating non-durable subscription at msg id -1:-1:-1:-1 - {}
2023-05-24T15:30:44,864 - INFO - [PulsarTestContext-executor-OrderedExecutor-0-0:NonDurableCursorImpl@66] - [prop-xyz2/ns1/persistent/__change_events-partition-4] Created non-durable cursor read-position=67:0 mark-delete-position=67:-1
2023-05-24T15:30:44,864 - INFO - [PulsarTestContext-executor-OrderedExecutor-0-0:ManagedLedgerImpl@1130] - [prop-xyz2/ns1/persistent/__change_events-partition-4] Opened new cursor: NonDurableCursorImpl{ledger=prop-xyz2/ns1/persistent/__change_events-partition-4, ackPos=67:-1, readPos=67:0}
2023-05-24T15:30:44,865 - INFO - [PulsarTestContext-executor-OrderedExecutor-0-0:AbstractDispatcherSingleActiveConsumer@211] - Removing consumer Consumer{subscription=PersistentSubscription{topic=persistent://prop-xyz2/ns1/__change_events-partition-4, name=multiTopicsReader-1b06c712cc}, consumerId=4, consumerName=ae38e, address=/127.0.0.1:57364}
2023-05-24T15:30:44,865 - INFO - [PulsarTestContext-executor-OrderedExecutor-0-0:PersistentSubscription@880] - [persistent://prop-xyz2/ns1/__change_events-partition-4][multiTopicsReader-1b06c712cc] Successfully closed subscription [NonDurableCursorImpl{ledger=prop-xyz2/ns1/persistent/__change_events-partition-4, ackPos=67:-1, readPos=67:0}]
2023-05-24T15:30:44,865 - INFO - [PulsarTestContext-executor-OrderedExecutor-0-0:PersistentSubscription@312] - [persistent://prop-xyz2/ns1/__change_events-partition-4][multiTopicsReader-1b06c712cc] Successfully closed dispatcher for reader
2023-05-24T15:30:44,865 - ERROR - [PulsarTestContext-executor-OrderedExecutor-0-0:PersistentTopic@948] - [persistent://prop-xyz2/ns1/__change_events-partition-4] Failed to create subscription: multiTopicsReader-1b06c712cc
java.util.concurrent.CompletionException: org.apache.pulsar.broker.service.BrokerServiceException: Connection was closed while the opening the cursor
at java.util.concurrent.CompletableFuture.encodeRelay(CompletableFuture.java:368) ~[?:?]
at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1189) ~[?:?]
at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2309) ~[?:?]
at org.apache.pulsar.broker.service.persistent.PersistentTopic.lambda$internalSubscribe$20(PersistentTopic.java:890) ~[classes/:?]
at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1187) ~[?:?]
at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2309) ~[?:?]
at org.apache.pulsar.broker.service.persistent.PersistentTopic.lambda$internalSubscribe$22(PersistentTopic.java:885) ~[classes/:?]
at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1187) ~[?:?]
at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2309) ~[?:?]
at org.apache.pulsar.broker.service.persistent.PersistentTopic.internalSubscribe(PersistentTopic.java:806) ~[classes/:?]
at org.apache.pulsar.broker.service.persistent.PersistentTopic.subscribe(PersistentTopic.java:780) ~[classes/:?]
at org.apache.pulsar.broker.service.ServerCnx.lambda$handleSubscribe$14(ServerCnx.java:1239) ~[classes/:?]
at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147) ~[?:?]
at org.apache.pulsar.broker.service.schema.BookkeeperSchemaStorage.lambda$closeLedger$42(BookkeeperSchemaStorage.java:619) ~[classes/:?]
at org.apache.bookkeeper.client.PulsarMockLedgerHandle.lambda$asyncClose$1(PulsarMockLedgerHandle.java:96) ~[testmocks-3.1.0-SNAPSHOT.jar:4.16.1]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?]
at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482) ~[?:?]
at org.apache.bookkeeper.common.util.SingleThreadExecutor.safeRunTask(SingleThreadExecutor.java:137) ~[bookkeeper-common-4.16.1.jar:4.16.1]
at org.apache.bookkeeper.common.util.SingleThreadExecutor.run(SingleThreadExecutor.java:107) ~[bookkeeper-common-4.16.1.jar:4.16.1]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.89.Final.jar:4.1.89.Final]
at java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: org.apache.pulsar.broker.service.BrokerServiceException: Connection was closed while the opening the cursor
at org.apache.pulsar.broker.service.persistent.PersistentTopic.lambda$internalSubscribe$19(PersistentTopic.java:914) ~[classes/:?]
at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1187) ~[?:?]
... 22 more
2023-05-24T15:30:44,866 - WARN - [PulsarTestContext-executor-OrderedExecutor-0-0:ServerCnx@1282] - [/127.0.0.1:57364][persistent://prop-xyz2/ns1/__change_events-partition-4][multiTopicsReader-1b06c712cc] Failed to create consumer: consumerId=4, Connection was closed while the opening the cursor
2023-05-24T15:30:44,922 - INFO - [broker-client-shared-timer-2719-1:ConnectionHandler@124] - [persistent://prop-xyz2/ns1/__change_events-partition-19] [multiTopicsReader-1b06c712cc] Reconnecting after connection was closed
2023-05-24T15:30:44,922 - INFO - [broker-client-shared-timer-2719-1:ConnectionHandler@68] - [persistent://prop-xyz2/ns1/__change_events-partition-19] [multiTopicsReader-1b06c712cc] Ignoring reconnection request (state: Closed)
2023-05-24T15:30:44,923 - INFO - [broker-client-shared-timer-2719-1:ConnectionHandler@124] - [persistent://prop-xyz2/ns1/__change_events-partition-17] [multiTopicsReader-1b06c712cc] Reconnecting after connection was closed
2023-05-24T15:30:44,923 - INFO - [broker-client-shared-timer-2719-1:ConnectionHandler@68] - [persistent://prop-xyz2/ns1/__change_events-partition-17] [multiTopicsReader-1b06c712cc] Ignoring reconnection request (state: Closed)
```
I wonder if the issue is related to how system topics are deleted?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on issue #20376: Flaky-test: AdminApi2Test.resetClusters
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on issue #20376:
URL: https://github.com/apache/pulsar/issues/20376#issuecomment-1564660663
Problem remains after the recent fix. reopening.
https://github.com/apache/pulsar/actions/runs/5092005952/jobs/9153089707?pr=20411#step:10:1446
```
Error: Tests run: 82, Failures: 1, Errors: 0, Skipped: 55, Time elapsed: 721.942 s <<< FAILURE! - in org.apache.pulsar.broker.admin.AdminApi2Test
Error: org.apache.pulsar.broker.admin.AdminApi2Test.resetClusters Time elapsed: 1.704 s <<< FAILURE!
org.apache.pulsar.client.admin.PulsarAdminException$ServerSideErrorException:
--- An unexpected error occurred in the server ---
Message: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /managed-ledgers/prop-xyz2
Stacktrace:
org.apache.pulsar.metadata.api.MetadataStoreException: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /managed-ledgers/prop-xyz2
at org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:476)
at org.apache.pulsar.metadata.impl.ZKMetadataStore.handleDeleteResult(ZKMetadataStore.java:304)
at org.apache.pulsar.metadata.impl.ZKMetadataStore.lambda$batchOperation$5(ZKMetadataStore.java:216)
at org.apache.zookeeper.MockZooKeeper.multi(MockZooKeeper.java:1006)
at org.apache.zookeeper.MockZooKeeperSession.multi(MockZooKeeperSession.java:191)
at org.apache.pulsar.metadata.impl.ZKMetadataStore.batchOperation(ZKMetadataStore.java:190)
at org.apache.pulsar.metadata.impl.batching.AbstractBatchedMetadataStore.internalBatchOperation(AbstractBatchedMetadataStore.java:184)
at org.apache.pulsar.metadata.impl.batching.AbstractBatchedMetadataStore.flush(AbstractBatchedMetadataStore.java:124)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /managed-ledgers/prop-xyz2
at org.apache.zookeeper.KeeperException.create(KeeperException.java:132)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
at org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:466)
... 14 more
at org.apache.pulsar.client.admin.PulsarAdminException.wrap(PulsarAdminException.java:252)
at org.apache.pulsar.client.admin.internal.BaseResource.sync(BaseResource.java:352)
at org.apache.pulsar.client.admin.internal.TenantsImpl.deleteTenant(TenantsImpl.java:93)
at org.apache.pulsar.broker.admin.AdminApi2Test.resetClusters(AdminApi2Test.java:181)
at jdk.internal.reflect.GeneratedMethodAccessor413.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.testng.internal.invokers.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:139)
at org.testng.internal.invokers.MethodInvocationHelper.invokeMethodConsideringTimeout(MethodInvocationHelper.java:69)
at org.testng.internal.invokers.ConfigInvoker.invokeConfigurationMethod(ConfigInvoker.java:361)
at org.testng.internal.invokers.ConfigInvoker.invokeConfigurations(ConfigInvoker.java:296)
at org.testng.internal.invokers.TestInvoker.runConfigMethods(TestInvoker.java:823)
at org.testng.internal.invokers.TestInvoker.runAfterConfigurations(TestInvoker.java:792)
at org.testng.internal.invokers.TestInvoker.invokeMethod(TestInvoker.java:768)
at org.testng.internal.invokers.TestInvoker.invokeTestMethod(TestInvoker.java:221)
at org.testng.internal.invokers.MethodRunner.runInSequence(MethodRunner.java:50)
at org.testng.internal.invokers.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:969)
at org.testng.internal.invokers.TestInvoker.invokeTestMethods(TestInvoker.java:194)
at org.testng.internal.invokers.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:148)
at org.testng.internal.invokers.TestMethodWorker.run(TestMethodWorker.java:128)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at org.testng.TestRunner.privateRun(TestRunner.java:829)
at org.testng.TestRunner.run(TestRunner.java:602)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:437)
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:431)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:391)
at org.testng.SuiteRunner.run(SuiteRunner.java:330)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:95)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1256)
at org.testng.TestNG.runSuitesLocally(TestNG.java:1176)
at org.testng.TestNG.runSuites(TestNG.java:1099)
at org.testng.TestNG.run(TestNG.java:1067)
at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:155)
at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeSingleClass(TestNGDirectoryTestSuite.java:102)
at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeLazy(TestNGDirectoryTestSuite.java:117)
at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:86)
at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:137)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:385)
at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:507)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:495)
Suppressed: org.apache.pulsar.client.admin.PulsarAdminException$ServerSideErrorException:
--- An unexpected error occurred in the server ---
Message: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /managed-ledgers/prop-xyz2
Stacktrace:
org.apache.pulsar.metadata.api.MetadataStoreException: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /managed-ledgers/prop-xyz2
at org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:476)
at org.apache.pulsar.metadata.impl.ZKMetadataStore.handleDeleteResult(ZKMetadataStore.java:304)
at org.apache.pulsar.metadata.impl.ZKMetadataStore.lambda$batchOperation$5(ZKMetadataStore.java:216)
at org.apache.zookeeper.MockZooKeeper.multi(MockZooKeeper.java:1006)
at org.apache.zookeeper.MockZooKeeperSession.multi(MockZooKeeperSession.java:191)
at org.apache.pulsar.metadata.impl.ZKMetadataStore.batchOperation(ZKMetadataStore.java:190)
at org.apache.pulsar.metadata.impl.batching.AbstractBatchedMetadataStore.internalBatchOperation(AbstractBatchedMetadataStore.java:184)
at org.apache.pulsar.metadata.impl.batching.AbstractBatchedMetadataStore.flush(AbstractBatchedMetadataStore.java:124)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /managed-ledgers/prop-xyz2
at org.apache.zookeeper.KeeperException.create(KeeperException.java:132)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
at org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:466)
... 14 more
at org.apache.pulsar.client.admin.internal.BaseResource.getApiException(BaseResource.java:272)
at org.apache.pulsar.client.admin.internal.BaseResource$4.failed(BaseResource.java:237)
at org.glassfish.jersey.client.JerseyInvocation$1.failed(JerseyInvocation.java:882)
at org.glassfish.jersey.client.JerseyInvocation$1.completed(JerseyInvocation.java:863)
at org.glassfish.jersey.client.ClientRuntime.processResponse(ClientRuntime.java:229)
at org.glassfish.jersey.client.ClientRuntime.access$200(ClientRuntime.java:62)
at org.glassfish.jersey.client.ClientRuntime$2.lambda$response$0(ClientRuntime.java:173)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:288)
at org.glassfish.jersey.client.ClientRuntime$2.response(ClientRuntime.java:173)
at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$apply$1(AsyncHttpConnector.java:254)
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$retryOperation$4(AsyncHttpConnector.java:296)
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
at org.asynchttpclient.netty.NettyResponseFuture.loadContent(NettyResponseFuture.java:222)
at org.asynchttpclient.netty.NettyResponseFuture.done(NettyResponseFuture.java:257)
at org.asynchttpclient.netty.handler.AsyncHttpClientHandler.finishUpdate(AsyncHttpClientHandler.java:241)
at org.asynchttpclient.netty.handler.HttpHandler.handleChunk(HttpHandler.java:114)
at org.asynchttpclient.netty.handler.HttpHandler.handleRead(HttpHandler.java:143)
at org.asynchttpclient.netty.handler.AsyncHttpClientHandler.channelRead(AsyncHttpClientHandler.java:78)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: javax.ws.rs.InternalServerErrorException: HTTP 500 {"reason":"\n --- An unexpected error occurred in the server ---\n\nMessage: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /managed-ledgers/prop-xyz2\n\nStacktrace:\n\norg.apache.pulsar.metadata.api.MetadataStoreException: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /managed-ledgers/prop-xyz2\n\tat org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:476)\n\tat org.apache.pulsar.metadata.impl.ZKMetadataStore.handleDeleteResult(ZKMetadataStore.java:304)\n\tat org.apache.pulsar.metadata.impl.ZKMetadataStore.lambda$batchOperation$5(ZKMetadataStore.java:216)\n\tat org.apache.zookeeper.MockZooKeeper.multi(MockZooKeeper.java:1006)\n\tat org.apache.zookeeper.MockZooKeeperSession.multi(MockZooKeeperSession.java:191)\n\tat org.apache.pulsar.metadata.impl.ZKMetadataStore.batchOperation(ZKMetada
taStore.java:190)\n\tat org.apache.pulsar.metadata.impl.batching.AbstractBatchedMetadataStore.internalBatchOperation(AbstractBatchedMetadataStore.java:184)\n\tat org.apache.pulsar.metadata.impl.batching.AbstractBatchedMetadataStore.flush(AbstractBatchedMetadataStore.java:124)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)\n\tat java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)\n\tat java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\nCaused by: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode =
Directory not empty for /managed-ledgers/prop-xyz2\n\tat org.apache.zookeeper.KeeperException.create(KeeperException.java:132)\n\tat org.apache.zookeeper.KeeperException.create(KeeperException.java:54)\n\tat org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:466)\n\t... 14 more\n"}
at org.glassfish.jersey.client.JerseyInvocation.convertToException(JerseyInvocation.java:960)
at org.glassfish.jersey.client.JerseyInvocation.access$700(JerseyInvocation.java:82)
... 54 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on issue #20376: Flaky-test: AdminApi2Test.resetClusters
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on issue #20376:
URL: https://github.com/apache/pulsar/issues/20376#issuecomment-1567104209
There are also this type of errors:
https://github.com/apache/pulsar/actions/runs/5111571116/jobs/9188966524?pr=20428#step:10:1449
```
Error: Tests run: 83, Failures: 1, Errors: 0, Skipped: 50, Time elapsed: 629.104 s <<< FAILURE! - in org.apache.pulsar.broker.admin.AdminApi2Test
Error: org.apache.pulsar.broker.admin.AdminApi2Test.resetClusters Time elapsed: 5.296 s <<< FAILURE!
org.testcontainers.shaded.org.awaitility.core.ConditionTimeoutException: Condition with org.apache.pulsar.broker.auth.MockedPulsarServiceBaseTest was not fulfilled within 5 seconds.
at org.testcontainers.shaded.org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:167)
at org.testcontainers.shaded.org.awaitility.core.CallableCondition.await(CallableCondition.java:78)
at org.testcontainers.shaded.org.awaitility.core.CallableCondition.await(CallableCondition.java:26)
at org.testcontainers.shaded.org.awaitility.core.ConditionFactory.until(ConditionFactory.java:985)
at org.testcontainers.shaded.org.awaitility.core.ConditionFactory.until(ConditionFactory.java:954)
at org.apache.pulsar.broker.auth.MockedPulsarServiceBaseTest.deleteNamespaceWithRetry(MockedPulsarServiceBaseTest.java:618)
at org.apache.pulsar.broker.auth.MockedPulsarServiceBaseTest.deleteNamespaceWithRetry(MockedPulsarServiceBaseTest.java:609)
at org.apache.pulsar.broker.admin.AdminApi2Test.resetClusters(AdminApi2Test.java:178)
at jdk.internal.reflect.GeneratedMethodAccessor413.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.testng.internal.invokers.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:139)
at org.testng.internal.invokers.MethodInvocationHelper.invokeMethodConsideringTimeout(MethodInvocationHelper.java:69)
at org.testng.internal.invokers.ConfigInvoker.invokeConfigurationMethod(ConfigInvoker.java:361)
at org.testng.internal.invokers.ConfigInvoker.invokeConfigurations(ConfigInvoker.java:296)
at org.testng.internal.invokers.TestInvoker.runConfigMethods(TestInvoker.java:823)
at org.testng.internal.invokers.TestInvoker.runAfterConfigurations(TestInvoker.java:792)
at org.testng.internal.invokers.TestInvoker.invokeMethod(TestInvoker.java:768)
at org.testng.internal.invokers.TestInvoker.invokeTestMethod(TestInvoker.java:221)
at org.testng.internal.invokers.MethodRunner.runInSequence(MethodRunner.java:50)
at org.testng.internal.invokers.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:969)
at org.testng.internal.invokers.TestInvoker.invokeTestMethods(TestInvoker.java:194)
at org.testng.internal.invokers.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:148)
at org.testng.internal.invokers.TestMethodWorker.run(TestMethodWorker.java:128)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at org.testng.TestRunner.privateRun(TestRunner.java:829)
at org.testng.TestRunner.run(TestRunner.java:602)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:437)
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:431)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:391)
at org.testng.SuiteRunner.run(SuiteRunner.java:330)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:95)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1256)
at org.testng.TestNG.runSuitesLocally(TestNG.java:1176)
at org.testng.TestNG.runSuites(TestNG.java:1099)
at org.testng.TestNG.run(TestNG.java:1067)
at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:155)
at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeSingleClass(TestNGDirectoryTestSuite.java:102)
at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeLazy(TestNGDirectoryTestSuite.java:117)
at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:86)
at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:137)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:385)
at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:507)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:495)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Flaky-test: AdminApi2Test.resetClusters [pulsar]
Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat closed issue #20376: Flaky-test: AdminApi2Test.resetClusters
URL: https://github.com/apache/pulsar/issues/20376
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] nicoloboschi closed issue #20376: Flaky-test: AdminApi2Test.resetClusters
Posted by "nicoloboschi (via GitHub)" <gi...@apache.org>.
nicoloboschi closed issue #20376: Flaky-test: AdminApi2Test.resetClusters
URL: https://github.com/apache/pulsar/issues/20376
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michaeljmarshall commented on issue #20376: Flaky-test: AdminApi2Test.resetClusters
Posted by "michaeljmarshall (via GitHub)" <gi...@apache.org>.
michaeljmarshall commented on issue #20376:
URL: https://github.com/apache/pulsar/issues/20376#issuecomment-1561507089
I downloaded the logs for this test, and here is some additional context for the error:
```
2023-05-24T15:30:44,959 - INFO - [configuration-metadata-store-2710-1:ResourceGroupNamespaceConfigListener@147] - Metadata store notification: Path /admin/policies/prop-xyz/ns1, Type Deleted
2023-05-24T15:30:44,960 - INFO - [metadata-store-2708-1:Namespaces@195] - [null] Successful delete namespace ns1
2023-05-24T15:30:44,960 - INFO - [metadata-store-2708-1:Slf4jRequestLogWriter@62] - 127.0.0.1 - - [24/May/2023:15:30:44 +0000] "DELETE /admin/v2/namespaces/prop-xyz/ns1?force=true HTTP/1.1" 204 0 "-" "Pulsar-Java-v3.1.0-SNAPSHOT" 30
2023-05-24T15:30:44,961 - INFO - [metadata-store-2708-1:NamespaceBundleFactory@183] - Policy updated for namespace Optional[prop-xyz/ns1], refreshing the bundle cache.
2023-05-24T15:30:44,970 - INFO - [metadata-store-2708-1:TenantsBase@205] - [null] Deleted tenant prop-xyz
2023-05-24T15:30:44,970 - INFO - [metadata-store-2708-1:Slf4jRequestLogWriter@62] - 127.0.0.1 - - [24/May/2023:15:30:44 +0000] "DELETE /admin/v2/tenants/prop-xyz?force=true HTTP/1.1" 204 0 "-" "Pulsar-Java-v3.1.0-SNAPSHOT" 5
2023-05-24T15:30:44,973 - INFO - [pulsar-web-2736-1:Slf4jRequestLogWriter@62] - 127.0.0.1 - - [24/May/2023:15:30:44 +0000] "GET /admin/v2/namespaces/prop-xyz2 HTTP/1.1" 200 2 "-" "Pulsar-Java-v3.1.0-SNAPSHOT" 1
2023-05-24T15:30:44,982 - ERROR - [metadata-store-2708-1:TenantsBase@210] - [null] Failed to delete tenant prop-xyz2
org.apache.pulsar.metadata.api.MetadataStoreException: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /managed-ledgers/prop-xyz2
at org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:476) ~[pulsar-metadata-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
at org.apache.pulsar.metadata.impl.ZKMetadataStore.handleDeleteResult(ZKMetadataStore.java:304) ~[pulsar-metadata-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
at org.apache.pulsar.metadata.impl.ZKMetadataStore.lambda$batchOperation$5(ZKMetadataStore.java:216) ~[pulsar-metadata-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
at org.apache.zookeeper.MockZooKeeper.multi(MockZooKeeper.java:1006) ~[testmocks-3.1.0-SNAPSHOT.jar:3.8.1]
at org.apache.zookeeper.MockZooKeeperSession.multi(MockZooKeeperSession.java:191) ~[testmocks-3.1.0-SNAPSHOT.jar:3.8.1]
at org.apache.pulsar.metadata.impl.ZKMetadataStore.batchOperation(ZKMetadataStore.java:190) ~[pulsar-metadata-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
at org.apache.pulsar.metadata.impl.batching.AbstractBatchedMetadataStore.internalBatchOperation(AbstractBatchedMetadataStore.java:184) ~[pulsar-metadata-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
at org.apache.pulsar.metadata.impl.batching.AbstractBatchedMetadataStore.flush(AbstractBatchedMetadataStore.java:124) ~[pulsar-metadata-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[?:?]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) ~[?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.89.Final.jar:4.1.89.Final]
at java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /managed-ledgers/prop-xyz2
at org.apache.zookeeper.KeeperException.create(KeeperException.java:132) ~[zookeeper-3.8.1.jar:3.8.1]
at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[zookeeper-3.8.1.jar:3.8.1]
at org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:466) ~[pulsar-metadata-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
... 14 more
```
Note that `resetClusters` is not a test, but is a lifecycle method that is called after every test run:
https://github.com/apache/pulsar/blob/e956db729f5098ff319237fd4220dbfb234c1b18/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/AdminApi2Test.java#L172C1-L190
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on issue #20376: Flaky-test: AdminApi2Test.resetClusters
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on issue #20376:
URL: https://github.com/apache/pulsar/issues/20376#issuecomment-1576679283
recent logs: https://gist.github.com/lhotari/4f71e2ce161a2dc6eb4f0d1f9543e1c0
happened here: https://github.com/apache/pulsar/actions/runs/5174095082/jobs/9320242188#step:10:1155
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] dave2wave commented on issue #20376: Flaky-test: AdminApi2Test.resetClusters
Posted by "dave2wave (via GitHub)" <gi...@apache.org>.
dave2wave commented on issue #20376:
URL: https://github.com/apache/pulsar/issues/20376#issuecomment-1625983842
This error is still blocking builds:
```
11:19:22 [ERROR] Tests run: 105, Failures: 1, Errors: 0, Skipped: 22, Time elapsed: 517.963 s <<< FAILURE! - in org.apache.pulsar.client.api.SimpleProducerConsumerTest
11:19:22 [ERROR] org.apache.pulsar.client.api.SimpleProducerConsumerTest.rest Time elapsed: 7.263 s <<< FAILURE!
11:19:22 org.apache.pulsar.client.admin.PulsarAdminException$ServerSideErrorException:
11:19:22
11:19:22 --- An unexpected error occurred in the server ---
11:19:22
11:19:22 Message: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /managed-ledgers/my-property
11:19:22
11:19:22 Stacktrace:
11:19:22
11:19:22 org.apache.pulsar.metadata.api.MetadataStoreException: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /managed-ledgers/my-property
11:19:22 at org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:476)
11:19:22 at org.apache.pulsar.metadata.impl.ZKMetadataStore.handleDeleteResult(ZKMetadataStore.java:304)
11:19:22 at org.apache.pulsar.metadata.impl.ZKMetadataStore.lambda$batchOperation$5(ZKMetadataStore.java:216)
11:19:22 at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
11:19:22 at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
11:19:22 at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
11:19:22 at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
11:19:22 at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
11:19:22 at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
11:19:22 at java.base/java.lang.Thread.run(Thread.java:833)
11:19:22 Caused by: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /managed-ledgers/my-property
11:19:22 at org.apache.zookeeper.KeeperException.create(KeeperException.java:132)
11:19:22 at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
11:19:22 at org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:466)
11:19:22 ... 9 more
11:19:22
11:19:22 at org.apache.pulsar.client.admin.PulsarAdminException.wrap(PulsarAdminException.java:252)
11:19:22 at org.apache.pulsar.client.admin.internal.BaseResource.sync(BaseResource.java:352)
11:19:22 at org.apache.pulsar.client.admin.internal.TenantsImpl.deleteTenant(TenantsImpl.java:93)
11:19:22 at org.apache.pulsar.client.api.SimpleProducerConsumerTest.rest(SimpleProducerConsumerTest.java:149)
11:19:22 at jdk.internal.reflect.GeneratedMethodAccessor340.invoke(Unknown Source)
11:19:22 at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
11:19:22 at java.base/java.lang.reflect.Method.invoke(Method.java:568)
11:19:22 at org.testng.internal.invokers.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:139)
11:19:22 at org.testng.internal.invokers.MethodInvocationHelper.invokeMethodConsideringTimeout(MethodInvocationHelper.java:69)
11:19:22 at org.testng.internal.invokers.ConfigInvoker.invokeConfigurationMethod(ConfigInvoker.java:361)
11:19:22 at org.testng.internal.invokers.ConfigInvoker.invokeConfigurations(ConfigInvoker.java:296)
11:19:22 at org.testng.internal.invokers.TestInvoker.runConfigMethods(TestInvoker.java:823)
11:19:22 at org.testng.internal.invokers.TestInvoker.runAfterConfigurations(TestInvoker.java:792)
11:19:22 at org.testng.internal.invokers.TestInvoker.invokeMethod(TestInvoker.java:768)
11:19:22 at org.testng.internal.invokers.TestInvoker.invokeTestMethod(TestInvoker.java:221)
11:19:22 at org.testng.internal.invokers.MethodRunner.runInSequence(MethodRunner.java:50)
11:19:22 at org.testng.internal.invokers.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:969)
11:19:22 at org.testng.internal.invokers.TestInvoker.invokeTestMethods(TestInvoker.java:194)
11:19:22 at org.testng.internal.invokers.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:148)
11:19:22 at org.testng.internal.invokers.TestMethodWorker.run(TestMethodWorker.java:128)
11:19:22 at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
11:19:22 at org.testng.TestRunner.privateRun(TestRunner.java:829)
11:19:22 at org.testng.TestRunner.run(TestRunner.java:602)
11:19:22 at org.testng.SuiteRunner.runTest(SuiteRunner.java:437)
11:19:22 at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:431)
11:19:22 at org.testng.SuiteRunner.privateRun(SuiteRunner.java:391)
11:19:22 at org.testng.SuiteRunner.run(SuiteRunner.java:330)
11:19:22 at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
11:19:22 at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:95)
11:19:22 at org.testng.TestNG.runSuitesSequentially(TestNG.java:1256)
11:19:22 at org.testng.TestNG.runSuitesLocally(TestNG.java:1176)
11:19:22 at org.testng.TestNG.runSuites(TestNG.java:1099)
11:19:22 at org.testng.TestNG.run(TestNG.java:1067)
11:19:22 at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:155)
11:19:22 at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeSingleClass(TestNGDirectoryTestSuite.java:102)
11:19:22 at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeLazy(TestNGDirectoryTestSuite.java:117)
11:19:22 at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:86)
11:19:22 at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:137)
11:19:22 at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:385)
11:19:22 at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
11:19:22 at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:507)
11:19:22 at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:495)
11:19:22 Suppressed: org.apache.pulsar.client.admin.PulsarAdminException$ServerSideErrorException:
11:19:22 --- An unexpected error occurred in the server ---
11:19:22
11:19:22 Message: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /managed-ledgers/my-property
11:19:22
11:19:22 Stacktrace:
11:19:22
11:19:22 org.apache.pulsar.metadata.api.MetadataStoreException: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /managed-ledgers/my-property
11:19:22 at org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:476)
11:19:22 at org.apache.pulsar.metadata.impl.ZKMetadataStore.handleDeleteResult(ZKMetadataStore.java:304)
11:19:22 at org.apache.pulsar.metadata.impl.ZKMetadataStore.lambda$batchOperation$5(ZKMetadataStore.java:216)
11:19:22 at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
11:19:22 at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
11:19:22 at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
11:19:22 at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
11:19:22 at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
11:19:22 at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
11:19:22 at java.base/java.lang.Thread.run(Thread.java:833)
11:19:22 Caused by: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /managed-ledgers/my-property
11:19:22 at org.apache.zookeeper.KeeperException.create(KeeperException.java:132)
11:19:22 at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
11:19:22 at org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:466)
11:19:22 ... 9 more
11:19:22
11:19:22 at org.apache.pulsar.client.admin.internal.BaseResource.getApiException(BaseResource.java:272)
11:19:22 at org.apache.pulsar.client.admin.internal.BaseResource$4.failed(BaseResource.java:237)
11:19:22 at org.glassfish.jersey.client.JerseyInvocation$1.failed(JerseyInvocation.java:882)
11:19:22 at org.glassfish.jersey.client.JerseyInvocation$1.completed(JerseyInvocation.java:863)
11:19:22 at org.glassfish.jersey.client.ClientRuntime.processResponse(ClientRuntime.java:229)
11:19:22 at org.glassfish.jersey.client.ClientRuntime.access$200(ClientRuntime.java:62)
11:19:22 at org.glassfish.jersey.client.ClientRuntime$2.lambda$response$0(ClientRuntime.java:173)
11:19:22 at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
11:19:22 at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
11:19:22 at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
11:19:22 at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
11:19:22 at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
11:19:22 at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:288)
11:19:22 at org.glassfish.jersey.client.ClientRuntime$2.response(ClientRuntime.java:173)
11:19:22 at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$apply$1(AsyncHttpConnector.java:254)
11:19:22 at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
11:19:22 at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
11:19:22 at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
11:19:22 at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
11:19:22 at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$retryOperation$4(AsyncHttpConnector.java:296)
11:19:22 at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
11:19:22 at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
11:19:22 at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
11:19:22 at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
11:19:22 at org.asynchttpclient.netty.NettyResponseFuture.loadContent(NettyResponseFuture.java:222)
11:19:22 at org.asynchttpclient.netty.NettyResponseFuture.done(NettyResponseFuture.java:257)
11:19:22 at org.asynchttpclient.netty.handler.AsyncHttpClientHandler.finishUpdate(AsyncHttpClientHandler.java:241)
11:19:22 at org.asynchttpclient.netty.handler.HttpHandler.handleChunk(HttpHandler.java:114)
11:19:22 at org.asynchttpclient.netty.handler.HttpHandler.handleRead(HttpHandler.java:143)
11:19:22 at org.asynchttpclient.netty.handler.AsyncHttpClientHandler.channelRead(AsyncHttpClientHandler.java:78)
11:19:22 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
11:19:22 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
11:19:22 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
11:19:22 at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
11:19:22 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
11:19:22 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
11:19:22 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
11:19:22 at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
11:19:22 at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
11:19:22 at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
11:19:22 at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
11:19:22 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
11:19:22 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
11:19:22 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
11:19:22 at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
11:19:22 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
11:19:22 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
11:19:22 at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
11:19:22 at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
11:19:22 at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
11:19:22 at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
11:19:22 at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
11:19:22 at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
11:19:22 at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
11:19:22 at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
11:19:22 at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
11:19:22 at java.base/java.lang.Thread.run(Thread.java:833)
11:19:22 Caused by: javax.ws.rs.InternalServerErrorException: HTTP 500 {"reason":"\n --- An unexpected error occurred in the server ---\n\nMessage: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /managed-ledgers/my-property\n\nStacktrace:\n\norg.apache.pulsar.metadata.api.MetadataStoreException: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /managed-ledgers/my-property\n\tat org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:476)\n\tat org.apache.pulsar.metadata.impl.ZKMetadataStore.handleDeleteResult(ZKMetadataStore.java:304)\n\tat org.apache.pulsar.metadata.impl.ZKMetadataStore.lambda$batchOperation$5(ZKMetadataStore.java:216)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base/java.util.concurrent.ScheduledThreadPoolExecu
tor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\nCaused by: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /managed-ledgers/my-property\n\tat org.apache.zookeeper.KeeperException.create(KeeperException.java:132)\n\tat org.apache.zookeeper.KeeperException.create(KeeperException.java:54)\n\tat org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:466)\n\t... 9 more\n"}
11:19:22 at org.glassfish.jersey.client.JerseyInvocation.convertToException(JerseyInvocation.java:960)
11:19:22 at org.glassfish.jersey.client.JerseyInvocation.access$700(JerseyInvocation.java:82)
11:19:22 ... 54 more
11:19:22 Caused by: [CIRCULAR REFERENCE: javax.ws.rs.InternalServerErrorException: HTTP 500 {"reason":"\n --- An unexpected error occurred in the server ---\n\nMessage: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /managed-ledgers/my-property\n\nStacktrace:\n\norg.apache.pulsar.metadata.api.MetadataStoreException: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /managed-ledgers/my-property\n\tat org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:476)\n\tat org.apache.pulsar.metadata.impl.ZKMetadataStore.handleDeleteResult(ZKMetadataStore.java:304)\n\tat org.apache.pulsar.metadata.impl.ZKMetadataStore.lambda$batchOperation$5(ZKMetadataStore.java:216)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base/java.util.concurrent.Sche
duledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\nCaused by: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /managed-ledgers/my-property\n\tat org.apache.zookeeper.KeeperException.create(KeeperException.java:132)\n\tat org.apache.zookeeper.KeeperException.create(KeeperException.java:54)\n\tat org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:466)\n\t... 9 more\n"}]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michaeljmarshall commented on issue #20376: Flaky-test: AdminApi2Test.resetClusters
Posted by "michaeljmarshall (via GitHub)" <gi...@apache.org>.
michaeljmarshall commented on issue #20376:
URL: https://github.com/apache/pulsar/issues/20376#issuecomment-1561515449
I wonder if there is a race condition for enabling force deletion of tenants?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michaeljmarshall commented on issue #20376: Flaky-test: AdminApi2Test.resetClusters
Posted by "michaeljmarshall (via GitHub)" <gi...@apache.org>.
michaeljmarshall commented on issue #20376:
URL: https://github.com/apache/pulsar/issues/20376#issuecomment-1561596147
Is it valid to have `persistent://prop-xyz2/ns1/__change_events-partition-11` as a persistent topic? This was recently fixed for another kind of system topic here https://github.com/apache/pulsar/pull/20370 by @heesung-sn.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michaeljmarshall commented on issue #20376: Flaky-test: AdminApi2Test.resetClusters
Posted by "michaeljmarshall (via GitHub)" <gi...@apache.org>.
michaeljmarshall commented on issue #20376:
URL: https://github.com/apache/pulsar/issues/20376#issuecomment-1577407119
I haven't been able to reproduce this locally. It would be really valuable to know the zk tree structure at this time of failure. I wonder if there is an easy way to get that so we can debug with more information. Perhaps we can enable zk event logs?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on issue #20376: Flaky-test: AdminApi2Test.resetClusters
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on issue #20376:
URL: https://github.com/apache/pulsar/issues/20376#issuecomment-1561345675
```
Error: Tests run: 104, Failures: 1, Errors: 0, Skipped: 61, Time elapsed: 465.204 s <<< FAILURE! - in org.apache.pulsar.client.api.SimpleProducerConsumerTest
Error: org.apache.pulsar.client.api.SimpleProducerConsumerTest.rest Time elapsed: 5.1 s <<< FAILURE!
org.apache.pulsar.client.admin.PulsarAdminException$ServerSideErrorException:
--- An unexpected error occurred in the server ---
Message: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /managed-ledgers/my-property
Stacktrace:
org.apache.pulsar.metadata.api.MetadataStoreException: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /managed-ledgers/my-property
at org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:476)
at org.apache.pulsar.metadata.impl.ZKMetadataStore.handleDeleteResult(ZKMetadataStore.java:304)
at org.apache.pulsar.metadata.impl.ZKMetadataStore.lambda$batchOperation$5(ZKMetadataStore.java:216)
at org.apache.zookeeper.MockZooKeeper.multi(MockZooKeeper.java:1006)
at org.apache.zookeeper.MockZooKeeperSession.multi(MockZooKeeperSession.java:191)
at org.apache.pulsar.metadata.impl.ZKMetadataStore.batchOperation(ZKMetadataStore.java:190)
at org.apache.pulsar.metadata.impl.batching.AbstractBatchedMetadataStore.internalBatchOperation(AbstractBatchedMetadataStore.java:184)
at org.apache.pulsar.metadata.impl.batching.AbstractBatchedMetadataStore.flush(AbstractBatchedMetadataStore.java:124)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /managed-ledgers/my-property
at org.apache.zookeeper.KeeperException.create(KeeperException.java:132)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
at org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:466)
... 14 more
at org.apache.pulsar.client.admin.PulsarAdminException.wrap(PulsarAdminException.java:252)
at org.apache.pulsar.client.admin.internal.BaseResource.sync(BaseResource.java:352)
at org.apache.pulsar.client.admin.internal.TenantsImpl.deleteTenant(TenantsImpl.java:93)
at org.apache.pulsar.client.api.SimpleProducerConsumerTest.rest(SimpleProducerConsumerTest.java:149)
at jdk.internal.reflect.GeneratedMethodAccessor338.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.testng.internal.invokers.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:139)
at org.testng.internal.invokers.MethodInvocationHelper.invokeMethodConsideringTimeout(MethodInvocationHelper.java:69)
at org.testng.internal.invokers.ConfigInvoker.invokeConfigurationMethod(ConfigInvoker.java:361)
at org.testng.internal.invokers.ConfigInvoker.invokeConfigurations(ConfigInvoker.java:296)
at org.testng.internal.invokers.TestInvoker.runConfigMethods(TestInvoker.java:823)
at org.testng.internal.invokers.TestInvoker.runAfterConfigurations(TestInvoker.java:792)
at org.testng.internal.invokers.TestInvoker.invokeMethod(TestInvoker.java:768)
at org.testng.internal.invokers.TestInvoker.invokeTestMethod(TestInvoker.java:221)
at org.testng.internal.invokers.MethodRunner.runInSequence(MethodRunner.java:50)
at org.testng.internal.invokers.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:969)
at org.testng.internal.invokers.TestInvoker.invokeTestMethods(TestInvoker.java:194)
at org.testng.internal.invokers.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:148)
at org.testng.internal.invokers.TestMethodWorker.run(TestMethodWorker.java:128)
at java.base/java.util.ArrayList.forEach(ArrayList.java:[1511](https://github.com/apache/pulsar/actions/runs/5068165357/jobs/9100493728#step:10:1512))
at org.testng.TestRunner.privateRun(TestRunner.java:829)
at org.testng.TestRunner.run(TestRunner.java:602)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:437)
at org.glassfish.jersey.client.JerseyInvocation$1.completed(JerseyInvocation.java:863)
at org.glassfish.jersey.client.ClientRuntime.processResponse(ClientRuntime.java:229)
at org.glassfish.jersey.client.ClientRuntime.access$200(ClientRuntime.java:62)
at org.glassfish.jersey.client.ClientRuntime$2.lambda$response$0(ClientRuntime.java:173)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:288)
at org.glassfish.jersey.client.ClientRuntime$2.response(ClientRuntime.java:173)
at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$apply$1(AsyncHttpConnector.java:254)
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$retryOperation$4(AsyncHttpConnector.java:296)
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
at org.asynchttpclient.netty.NettyResponseFuture.loadContent(NettyResponseFuture.java:222)
at org.asynchttpclient.netty.NettyResponseFuture.done(NettyResponseFuture.java:257)
at org.asynchttpclient.netty.handler.AsyncHttpClientHandler.finishUpdate(AsyncHttpClientHandler.java:241)
at org.asynchttpclient.netty.handler.HttpHandler.handleChunk(HttpHandler.java:114)
at org.asynchttpclient.netty.handler.HttpHandler.handleRead(HttpHandler.java:143)
at org.asynchttpclient.netty.handler.AsyncHttpClientHandler.channelRead(AsyncHttpClientHandler.java:78)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: javax.ws.rs.InternalServerErrorException: HTTP 500 {"reason":"\n --- An unexpected error occurred in the server ---\n\nMessage: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /managed-ledgers/my-property\n\nStacktrace:\n\norg.apache.pulsar.metadata.api.MetadataStoreException: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /managed-ledgers/my-property\n\tat org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:476)\n\tat org.apache.pulsar.metadata.impl.ZKMetadataStore.handleDeleteResult(ZKMetadataStore.java:304)\n\tat org.apache.pulsar.metadata.impl.ZKMetadataStore.lambda$batchOperation$5(ZKMetadataStore.java:216)\n\tat org.apache.zookeeper.MockZooKeeper.multi(MockZooKeeper.java:1006)\n\tat org.apache.zookeeper.MockZooKeeperSession.multi(MockZooKeeperSession.java:191)\n\tat org.apache.pulsar.metadata.impl.ZKMetadataStore.batchOperation(ZKMe
tadataStore.java:190)\n\tat org.apache.pulsar.metadata.impl.batching.AbstractBatchedMetadataStore.internalBatchOperation(AbstractBatchedMetadataStore.java:184)\n\tat org.apache.pulsar.metadata.impl.batching.AbstractBatchedMetadataStore.flush(AbstractBatchedMetadataStore.java:124)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)\n\tat java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)\n\tat java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\nCaused by: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCod
e = Directory not empty for /managed-ledgers/my-property\n\tat org.apache.zookeeper.KeeperException.create(KeeperException.java:132)\n\tat org.apache.zookeeper.KeeperException.create(KeeperException.java:54)\n\tat org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:466)\n\t... 14 more\n"}
at org.glassfish.jersey.client.JerseyInvocation.convertToException(JerseyInvocation.java:960)
at org.glassfish.jersey.client.JerseyInvocation.access$700(JerseyInvocation.java:82)
... 54 more
Caused by: [CIRCULAR REFERENCE: javax.ws.rs.InternalServerErrorException: HTTP 500 {"reason":"\n --- An unexpected error occurred in the server ---\n\nMessage: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /managed-ledgers/my-property\n\nStacktrace:\n\norg.apache.pulsar.metadata.api.MetadataStoreException: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /managed-ledgers/my-property\n\tat org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:476)\n\tat org.apache.pulsar.metadata.impl.ZKMetadataStore.handleDeleteResult(ZKMetadataStore.java:304)\n\tat org.apache.pulsar.metadata.impl.ZKMetadataStore.lambda$batchOperation$5(ZKMetadataStore.java:216)\n\tat org.apache.zookeeper.MockZooKeeper.multi(MockZooKeeper.java:1006)\n\tat org.apache.zookeeper.MockZooKeeperSession.multi(MockZooKeeperSession.java:191)\n\tat org.apache.pulsar.metadata.impl.ZKMetadataStore
.batchOperation(ZKMetadataStore.java:190)\n\tat org.apache.pulsar.metadata.impl.batching.AbstractBatchedMetadataStore.internalBatchOperation(AbstractBatchedMetadataStore.java:184)\n\tat org.apache.pulsar.metadata.impl.batching.AbstractBatchedMetadataStore.flush(AbstractBatchedMetadataStore.java:124)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)\n\tat java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)\n\tat java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\nCaused by: org.apache.zookeeper.KeeperException$NotEmptyExcep
tion: KeeperErrorCode = Directory not empty for /managed-ledgers/my-property\n\tat org.apache.zookeeper.KeeperException.create(KeeperException.java:132)\n\tat org.apache.zookeeper.KeeperException.create(KeeperException.java:54)\n\tat org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:466)\n\t... 14 more\n"}]
```
https://github.com/apache/pulsar/actions/runs/5068165357/jobs/9100493728#step:10:1495
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michaeljmarshall commented on issue #20376: Flaky-test: AdminApi2Test.resetClusters
Posted by "michaeljmarshall (via GitHub)" <gi...@apache.org>.
michaeljmarshall commented on issue #20376:
URL: https://github.com/apache/pulsar/issues/20376#issuecomment-1561543337
It seems likely that something is incorrectly coordinated. Further, it looks like tenant deletion can easily lead to corrupted metadata. The logic is just chained futures without error handling:
https://github.com/apache/pulsar/blob/dc29f3a4b6c900cf2003c70c266bf3ab62fde87d/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/TenantsBase.java#L224-L241
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michaeljmarshall commented on issue #20376: Flaky-test: AdminApi2Test.resetClusters
Posted by "michaeljmarshall (via GitHub)" <gi...@apache.org>.
michaeljmarshall commented on issue #20376:
URL: https://github.com/apache/pulsar/issues/20376#issuecomment-1561609427
It looks like the test intentionally creates partitioned topics for the `__change_events` topic, so that might be a non-issue here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on issue #20376: Flaky-test: AdminApi2Test.resetClusters
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on issue #20376:
URL: https://github.com/apache/pulsar/issues/20376#issuecomment-1564672710
Here's the logs for the failure case: https://gist.github.com/lhotari/92d7e2df3791c0684d3b043b8bbc0420
extracted from the Unit-BROKER_GROUP_3-surefire-reports.zip file with this command:
```
xmlstarlet sel -t -m '//testcase[@name="resetClusters"]' -v @classname -o '.' -v @name -n -v failure -n -v system-out -n pulsar-broker/target/surefire-reports/TEST-org.apache.pulsar.broker.admin.AdminApi2Test.xml
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org