You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/02/05 09:21:56 UTC
[GitHub] [pulsar] lichuan6 opened a new issue #9491: failed to delete topic with HTTP 500 Internal Server Error
lichuan6 opened a new issue #9491:
URL: https://github.com/apache/pulsar/issues/9491
**Describe the bug**
I have a producer which produces logs to `public/logging/app` and a consumer which consumes the messages and sinks to elaticsearch. Occasionally, for debug purpose, I delete the topic or namespace using `pulsar-admin topics delete public/logging/app --force` or `pulsar-admin namespaces delete public/logging --force`. The reason why I use `--force` is that I cannot delete them when there're still consumers or producers for the topic:
```
09:20:03.977 [AsyncHttpClient-7-1] WARN org.apache.pulsar.client.admin.internal.BaseResource - [http://pulsar-broker.pulsar:8080/admin/v2/persistent/public/logging/app?force=false&deleteSchema=false] Failed to perform http delete request: javax.ws.rs.ClientErrorException: HTTP 412 Precondition Failed
Topic has active producers/subscriptions
Reason: Topic has active producers/subscriptions
command terminated with exit code 1
```
After trying few times of force delete , I find out I can't delete topics any more:
```
08:56:53.184 [AsyncHttpClient-7-1] WARN org.apache.pulsar.client.admin.internal.BaseResource - [http://pulsar-broker.pulsar:8080/admin/v2/persistent/public/logging/app?force=true&deleteSchema=false] Failed to perform http delete request: javax.ws.rs.InternalServerErrorException: HTTP 500 Internal Server Error
HTTP 500 Internal Server Error
Reason: HTTP 500 Internal Server Error
command terminated with exit code 1
```
Here's the broker log
```
09:01:44.460 [pulsar-web-45-4] ERROR org.apache.pulsar.broker.admin.impl.PersistentTopicsBase - [admin] Failed to delete topic forcefully persistent://public/logging/app
java.util.concurrent.ExecutionException: org.apache.pulsar.broker.service.BrokerServiceException$TopicBusyException: Topic has -13 connected producers/consumers
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) ~[?:1.8.0_282]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908) ~[?:1.8.0_282]
at org.apache.pulsar.broker.admin.impl.PersistentTopicsBase.internalDeleteTopicForcefully(PersistentTopicsBase.java:374) ~[org.apache.pulsar-pulsar-broker-2.6.3.jar:2.6.3]
at org.apache.pulsar.broker.admin.impl.PersistentTopicsBase.internalDeleteTopic(PersistentTopicsBase.java:814) ~[org.apache.pulsar-pulsar-broker-2.6.3.jar:2.6.3]
at org.apache.pulsar.broker.admin.v2.PersistentTopics.deleteTopic(PersistentTopics.java:444) ~[org.apache.pulsar-pulsar-broker-2.6.3.jar:2.6.3]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_282]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_282]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_282]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_282]
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[org.glassfish.jersey.core-jersey-server-2.31.jar:?]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124) ~[org.glassfish.jersey.core-jersey-server-2.31.jar:?]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167) ~[org.glassfish.jersey.core-jersey-server-2.31.jar:?]
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$VoidOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:159) ~[org.glassfish.jersey.core-jersey-server-2.31.jar:?]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79) ~[org.glassfish.jersey.core-jersey-server-2.31.jar:?]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469) ~[org.glassfish.jersey.core-jersey-server-2.31.jar:?]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391) ~[org.glassfish.jersey.core-jersey-server-2.31.jar:?]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80) ~[org.glassfish.jersey.core-jersey-server-2.31.jar:?]
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:253) ~[org.glassfish.jersey.core-jersey-server-2.31.jar:?]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) ~[org.glassfish.jersey.core-jersey-common-2.31.jar:?]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) ~[org.glassfish.jersey.core-jersey-common-2.31.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[org.glassfish.jersey.core-jersey-common-2.31.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[org.glassfish.jersey.core-jersey-common-2.31.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:244) ~[org.glassfish.jersey.core-jersey-common-2.31.jar:?]
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) ~[org.glassfish.jersey.core-jersey-common-2.31.jar:?]
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232) ~[org.glassfish.jersey.core-jersey-server-2.31.jar:?]
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680) ~[org.glassfish.jersey.core-jersey-server-2.31.jar:?]
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394) ~[org.glassfish.jersey.containers-jersey-container-servlet-core-2.31.jar:?]
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346) ~[org.glassfish.jersey.containers-jersey-container-servlet-core-2.31.jar:?]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366) ~[org.glassfish.jersey.containers-jersey-container-servlet-core-2.31.jar:?]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319) ~[org.glassfish.jersey.containers-jersey-container-servlet-core-2.31.jar:?]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205) ~[org.glassfish.jersey.containers-jersey-container-servlet-core-2.31.jar:?]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:763) ~[org.eclipse.jetty-jetty-servlet-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1633) ~[org.eclipse.jetty-jetty-servlet-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.apache.pulsar.broker.web.ResponseHandlerFilter.doFilter(ResponseHandlerFilter.java:58) ~[org.apache.pulsar-pulsar-broker-2.6.3.jar:2.6.3]
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[org.eclipse.jetty-jetty-servlet-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1609) ~[org.eclipse.jetty-jetty-servlet-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.apache.pulsar.broker.web.AuthenticationFilter.doFilter(AuthenticationFilter.java:82) ~[org.apache.pulsar-pulsar-broker-common-2.6.3.jar:2.6.3]
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[org.eclipse.jetty-jetty-servlet-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1609) ~[org.eclipse.jetty-jetty-servlet-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:561) ~[org.eclipse.jetty-jetty-servlet-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[org.eclipse.jetty-jetty-server-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1612) ~[org.eclipse.jetty-jetty-server-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[org.eclipse.jetty-jetty-server-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434) ~[org.eclipse.jetty-jetty-server-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) ~[org.eclipse.jetty-jetty-server-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501) ~[org.eclipse.jetty-jetty-servlet-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1582) ~[org.eclipse.jetty-jetty-server-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) ~[org.eclipse.jetty-jetty-server-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349) ~[org.eclipse.jetty-jetty-server-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[org.eclipse.jetty-jetty-server-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:234) ~[org.eclipse.jetty-jetty-server-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146) ~[org.eclipse.jetty-jetty-server-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:179) ~[org.eclipse.jetty-jetty-server-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[org.eclipse.jetty-jetty-server-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.server.Server.handle(Server.java:516) ~[org.eclipse.jetty-jetty-server-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383) ~[org.eclipse.jetty-jetty-server-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556) [org.eclipse.jetty-jetty-server-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375) [org.eclipse.jetty-jetty-server-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273) [org.eclipse.jetty-jetty-server-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) [org.eclipse.jetty-jetty-io-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) [org.eclipse.jetty-jetty-io-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) [org.eclipse.jetty-jetty-io-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) [org.eclipse.jetty-jetty-util-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) [org.eclipse.jetty-jetty-util-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) [org.eclipse.jetty-jetty-util-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) [org.eclipse.jetty-jetty-util-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375) [org.eclipse.jetty-jetty-util-9.4.33.v20201020.jar:9.4.33.v20201020]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_282]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_282]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.48.Final.jar:4.1.48.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282]
Caused by: org.apache.pulsar.broker.service.BrokerServiceException$TopicBusyException: Topic has -13 connected producers/consumers
at org.apache.pulsar.broker.service.persistent.PersistentTopic.lambda$delete$22(PersistentTopic.java:940) ~[org.apache.pulsar-pulsar-broker-2.6.3.jar:2.6.3]
at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:670) ~[?:1.8.0_282]
at java.util.concurrent.CompletableFuture.uniAcceptStage(CompletableFuture.java:683) ~[?:1.8.0_282]
at java.util.concurrent.CompletableFuture.thenAccept(CompletableFuture.java:2010) ~[?:1.8.0_282]
at org.apache.pulsar.broker.service.persistent.PersistentTopic.delete(PersistentTopic.java:880) ~[org.apache.pulsar-pulsar-broker-2.6.3.jar:2.6.3]
at org.apache.pulsar.broker.service.persistent.PersistentTopic.deleteForcefully(PersistentTopic.java:829) ~[org.apache.pulsar-pulsar-broker-2.6.3.jar:2.6.3]
at org.apache.pulsar.broker.service.BrokerService.deleteTopic(BrokerService.java:777) ~[org.apache.pulsar-pulsar-broker-2.6.3.jar:2.6.3]
... 69 more
09:01:44.464 [pulsar-web-45-4] INFO org.eclipse.jetty.server.RequestLog - 172.24.170.210 - - [05/Feb/2021:09:01:44 +0000] "DELETE /admin/v2/persistent/public/logging/app?force=true&deleteSchema=false HTTP/1.1" 500 7353 "-" "Pulsar-Java-v2.6.3" 6
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lichuan6 commented on issue #9491: failed to delete topic with HTTP 500 Internal Server Error
Posted by GitBox <gi...@apache.org>.
lichuan6 commented on issue #9491:
URL: https://github.com/apache/pulsar/issues/9491#issuecomment-774867199
@codelipenghui
As it is not practical and takes much engineering time to upgrade the whole pulsar cluster, any workaround to fix this in the old version?
Thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] YanshuoH commented on issue #9491: failed to delete topic with HTTP 500 Internal Server Error
Posted by GitBox <gi...@apache.org>.
YanshuoH commented on issue #9491:
URL: https://github.com/apache/pulsar/issues/9491#issuecomment-775653246
@codelipenghui And do we have an ETA for the 2.6.4 release? Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #9491: failed to delete topic with HTTP 500 Internal Server Error
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on issue #9491:
URL: https://github.com/apache/pulsar/issues/9491#issuecomment-1058893451
The issue had no activity for 30 days, mark with Stale label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #9491: failed to delete topic with HTTP 500 Internal Server Error
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on issue #9491:
URL: https://github.com/apache/pulsar/issues/9491#issuecomment-774075642
@lichuan6 I think this PR https://github.com/apache/pulsar/pull/7356 is fixed this problem but 2.6.3 does not contain this fix. You can try the 2.7.0.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] hepyu commented on issue #9491: failed to delete topic with HTTP 500 Internal Server Error
Posted by GitBox <gi...@apache.org>.
hepyu commented on issue #9491:
URL: https://github.com/apache/pulsar/issues/9491#issuecomment-899240488
> Did this make it into 2.8.0? I am experiencing what looks like the same issue on that version.
same question on same version：2.7.0。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lichuan6 commented on issue #9491: failed to delete topic with HTTP 500 Internal Server Error
Posted by GitBox <gi...@apache.org>.
lichuan6 commented on issue #9491:
URL: https://github.com/apache/pulsar/issues/9491#issuecomment-774986755
My current solution was to remove data in zookeeper using `zdCli.sh`:
```bash
# in zookeeper pod
./zkCli.sh rmr /managed-ledgers/public/logging/persistent/app
```
The topic was deleted.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lichuan6 edited a comment on issue #9491: failed to delete topic with HTTP 500 Internal Server Error
Posted by GitBox <gi...@apache.org>.
lichuan6 edited a comment on issue #9491:
URL: https://github.com/apache/pulsar/issues/9491#issuecomment-774986755
My current solution was to remove the data in zookeeper using `zdCli.sh`:
```bash
# in zookeeper pod
./zkCli.sh rmr /managed-ledgers/public/logging/persistent/app
```
The topic was deleted.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #9491: failed to delete topic with HTTP 500 Internal Server Error
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on issue #9491:
URL: https://github.com/apache/pulsar/issues/9491#issuecomment-775636218
@lichuan6 I have marked #7356 for the 2.6.4 release. We might release a minor version 2.6.4 for fixing some issues that 2.6.3 have.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] zbentley commented on issue #9491: failed to delete topic with HTTP 500 Internal Server Error
Posted by GitBox <gi...@apache.org>.
zbentley commented on issue #9491:
URL: https://github.com/apache/pulsar/issues/9491#issuecomment-893037723
Did this make it into 2.8.0? I am experiencing what looks like the same issue on that version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #9491: failed to delete topic with HTTP 500 Internal Server Error
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on issue #9491:
URL: https://github.com/apache/pulsar/issues/9491#issuecomment-788962633
@YanshuoH We are releasing the 2.7.1 now, I think we can start a minor release for 2.6.4 after 2.7.1 released.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] zbentley commented on issue #9491: failed to delete topic with HTTP 500 Internal Server Error
Posted by GitBox <gi...@apache.org>.
zbentley commented on issue #9491:
URL: https://github.com/apache/pulsar/issues/9491#issuecomment-893037723
Did this make it into 2.8.0? I am experiencing what looks like the same issue on that version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org