You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2020/08/26 20:47:33 UTC
[GitHub] [pulsar] devinbost opened a new issue #7911: Unable to produce to specific topic after upgrading to 2.6.1
devinbost opened a new issue #7911:
URL: https://github.com/apache/pulsar/issues/7911
I ran into a very strange issue after upgrading to Pulsar 2.6.1. When upgrading, I basically rebuilt this cluster from scratch, so there shouldn't be any remnant of the previous cluster. However, I still had a producer and a consumer writing to and consuming from the topic "persistent://public/default/canary" as a health check. I didn't remove these containers until I finished rolling out the new cluster, but then when I restarted the canary producer and canary consumer, they couldn't connect to their topic. Other producers and consumers started just fine.
I was able to reproduce the issue by manually attempting to produce to this topic. When I produce to this topic, I get a broker timeout exception. However, if I try producing to any other topic, I don't get any exception.
See below.
`root@server26:/pulsar# bin/pulsar-client produce persistent://public/default/canary -m "TEST ME"`
> 2020-08-26T20:39:42,787 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.BinaryProtoLookupService - [persistent://public/default/canary] failed to send lookup request : org.apache.pulsar.client.api.PulsarClientException$TimeoutException: 2 lookup request timedout after ms 30000
> 2020-08-26T20:39:42,789 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.BinaryProtoLookupService - [persistent://public/default/canary] lookup failed : org.apache.pulsar.client.api.PulsarClientException$TimeoutException: 2 lookup request timedout after ms 30000
> java.util.concurrent.CompletionException: org.apache.pulsar.client.api.PulsarClientException$TimeoutException: 2 lookup request timedout after ms 30000
> at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) ~[?:1.8.0_252]
> at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) ~[?:1.8.0_252]
> at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:661) ~[?:1.8.0_252]
> at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:646) ~[?:1.8.0_252]
> at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) ~[?:1.8.0_252]
> at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) ~[?:1.8.0_252]
> at org.apache.pulsar.client.impl.ClientCnx.lambda$addPendingLookupRequests$12(ClientCnx.java:541) ~[org.apache.pulsar-pulsar-client-original-2.6.1.jar:2.6.1]
> at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) [io.netty-netty-common-4.1.48.Final.jar:4.1.48.Final]
> at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) [io.netty-netty-common-4.1.48.Final.jar:4.1.48.Final]
> at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) [io.netty-netty-common-4.1.48.Final.jar:4.1.48.Final]
> at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) [io.netty-netty-common-4.1.48.Final.jar:4.1.48.Final]
> at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) [io.netty-netty-transport-native-epoll-4.1.48.Final-linux-x86_64.jar:4.1.48.Final]
> at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [io.netty-netty-common-4.1.48.Final.jar:4.1.48.Final]
> at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [io.netty-netty-common-4.1.48.Final.jar:4.1.48.Final]
> at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.48.Final.jar:4.1.48.Final]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]
> Caused by: org.apache.pulsar.client.api.PulsarClientException$TimeoutException: 2 lookup request timedout after ms 30000
> ... 10 more
> 2020-08-26T20:39:42,794 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.ConnectionHandler - [persistent://public/default/canary] [null] Error connecting to broker: org.apache.pulsar.client.api.PulsarClientException$TimeoutException: 2 lookup request timedout after ms 30000
> 2020-08-26T20:39:42,794 [main] ERROR org.apache.pulsar.client.cli.PulsarClientTool - Error while producing messages
> 2020-08-26T20:39:42,795 [main] ERROR org.apache.pulsar.client.cli.PulsarClientTool - 2 lookup request timedout after ms 30000
> org.apache.pulsar.client.api.PulsarClientException$TimeoutException: 2 lookup request timedout after ms 30000
> at org.apache.pulsar.client.api.PulsarClientException.unwrap(PulsarClientException.java:856) ~[org.apache.pulsar-pulsar-client-api-2.6.1.jar:2.6.1]
> at org.apache.pulsar.client.impl.ProducerBuilderImpl.create(ProducerBuilderImpl.java:93) ~[org.apache.pulsar-pulsar-client-original-2.6.1.jar:2.6.1]
> at org.apache.pulsar.client.cli.CmdProduce.publish(CmdProduce.java:211) [org.apache.pulsar-pulsar-client-tools-2.6.1.jar:2.6.1]
> at org.apache.pulsar.client.cli.CmdProduce.run(CmdProduce.java:196) [org.apache.pulsar-pulsar-client-tools-2.6.1.jar:2.6.1]
> at org.apache.pulsar.client.cli.PulsarClientTool.run(PulsarClientTool.java:169) [org.apache.pulsar-pulsar-client-tools-2.6.1.jar:2.6.1]
> at org.apache.pulsar.client.cli.PulsarClientTool.main(PulsarClientTool.java:203) [org.apache.pulsar-pulsar-client-tools-2.6.1.jar:2.6.1]
`root@server26:/pulsar# bin/pulsar-client produce persistent://public/default/canary2 -m "TEST ME"`
[no exception]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] devinbost commented on issue #7911: Unable to produce to specific topic in /public/default after upgrading to 2.6.1
Posted by GitBox <gi...@apache.org>.
devinbost commented on issue #7911:
URL: https://github.com/apache/pulsar/issues/7911#issuecomment-681191394
It's like the topic is locked up in some sort of bizarre state where it can't be deleted (which also results in timeouts) or anything...
Is there someplace in Zookeeper where I might go manually look for more information about this topic?
Or, is there a way of getting more information from Bookkeeper directly about it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] devinbost commented on issue #7911: Unable to produce to specific topic in /public/default after upgrading to 2.6.1
Posted by GitBox <gi...@apache.org>.
devinbost commented on issue #7911:
URL: https://github.com/apache/pulsar/issues/7911#issuecomment-681136975
I don't know if it's related, but my Zookeeper nodes keep saying:
> 2020-08-26T21:35:15,270 [NIOWorkerThread-4] WARN org.apache.zookeeper.server.NIOServerCnxn - Unable to read additional data from client sessionid 0x0, likely client has closed socket
> 2020-08-26T21:35:17,204 [NIOWorkerThread-5] WARN org.apache.zookeeper.server.NIOServerCnxn - Unable to read additional data from client sessionid 0x0, likely client has closed socket
> 2020-08-26T21:35:25,265 [NIOWorkerThread-4] WARN org.apache.zookeeper.server.NIOServerCnxn - Unable to read additional data from client sessionid 0x0, likely client has closed socket
> 2020-08-26T21:35:27,200 [NIOWorkerThread-7] WARN org.apache.zookeeper.server.NIOServerCnxn - Unable to read additional data from client sessionid 0x0, likely client has closed socket
> 2020-08-26T21:35:35,260 [NIOWorkerThread-4] WARN org.apache.zookeeper.server.NIOServerCnxn - Unable to read additional data from client sessionid 0x0, likely client has closed socket
> 2020-08-26T21:35:37,195 [NIOWorkerThread-5] WARN org.apache.zookeeper.server.NIOServerCnxn - Unable to read additional data from client sessionid 0x0, likely client has closed socket
and I don't remember seeing that in their logs before.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] devinbost commented on issue #7911: Unable to produce to specific topic in /public/default after upgrading to 2.6.1
Posted by GitBox <gi...@apache.org>.
devinbost commented on issue #7911:
URL: https://github.com/apache/pulsar/issues/7911#issuecomment-681116559
Also, by changing the canary producer and canary consumer to use `persistent://public/default/canary1` instead of `persistent://public/default/canary`, they work just fine.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie commented on issue #7911: Unable to produce to specific topic in /public/default after upgrading to 2.6.1
Posted by GitBox <gi...@apache.org>.
sijie commented on issue #7911:
URL: https://github.com/apache/pulsar/issues/7911#issuecomment-682056696
@devinbost for such cases, a heap dump is appreciated to investigate the problem. Can you take a heap dump on the broker?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] devinbost commented on issue #7911: Unable to produce to specific topic in /public/default after upgrading to 2.6.1
Posted by GitBox <gi...@apache.org>.
devinbost commented on issue #7911:
URL: https://github.com/apache/pulsar/issues/7911#issuecomment-681174475
I found this exception on one of the brokers:
> 2020-08-26T23:23:12,142 [pulsar-web-45-7] ERROR org.apache.pulsar.broker.web.PulsarWebResource - Finding owner for topic persistent://public/default/canary timed out
> java.util.concurrent.TimeoutException: null
> at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1784) ~[?:1.8.0_252]
> at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928) ~[?:1.8.0_252]
> at org.apache.pulsar.broker.namespace.NamespaceService.getWebServiceUrl(NamespaceService.java:215) ~[org.apache.pulsar-pulsar-broker-2.6.1.jar:2.6.1]
> at org.apache.pulsar.broker.web.PulsarWebResource.validateTopicOwnership(PulsarWebResource.java:599) ~[org.apache.pulsar-pulsar-broker-2.6.1.jar:2.6.1]
> at org.apache.pulsar.broker.admin.impl.PersistentTopicsBase.internalGetInternalStats(PersistentTopicsBase.java:914) ~[org.apache.pulsar-pulsar-broker-2.6.1.jar:2.6.1]
> at org.apache.pulsar.broker.admin.v2.PersistentTopics.getInternalStats(PersistentTopics.java:516) ~[org.apache.pulsar-pulsar-broker-2.6.1.jar:2.6.1]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_252]
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_252]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_252]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_252]
> at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76) ~[org.glassfish.jersey.core-jersey-server-2.27.jar:?]
> at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148) ~[org.glassfish.jersey.core-jersey-server-2.27.jar:?]
> at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191) ~[org.glassfish.jersey.core-jersey-server-2.27.jar:?]
> at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:243) ~[org.glassfish.jersey.core-jersey-server-2.27.jar:?]
> at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103) ~[org.glassfish.jersey.core-jersey-server-2.27.jar:?]
> at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493) ~[org.glassfish.jersey.core-jersey-server-2.27.jar:?]
> at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415) ~[org.glassfish.jersey.core-jersey-server-2.27.jar:?]
> at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104) ~[org.glassfish.jersey.core-jersey-server-2.27.jar:?]
> at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277) ~[org.glassfish.jersey.core-jersey-server-2.27.jar:?]
> at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) ~[org.glassfish.jersey.core-jersey-common-2.27.jar:?]
> at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) ~[org.glassfish.jersey.core-jersey-common-2.27.jar:?]
> at org.glassfish.jersey.internal.Errors.process(Errors.java:316) ~[org.glassfish.jersey.core-jersey-common-2.27.jar:?]
> at org.glassfish.jersey.internal.Errors.process(Errors.java:298) ~[org.glassfish.jersey.core-jersey-common-2.27.jar:?]
> at org.glassfish.jersey.internal.Errors.process(Errors.java:268) ~[org.glassfish.jersey.core-jersey-common-2.27.jar:?]
> at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) ~[org.glassfish.jersey.core-jersey-common-2.27.jar:?]
> at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) ~[org.glassfish.jersey.core-jersey-server-2.27.jar:?]
> at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) ~[org.glassfish.jersey.core-jersey-server-2.27.jar:?]
> at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) ~[org.glassfish.jersey.containers-jersey-container-servlet-core-2.27.jar:?]
> at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) ~[org.glassfish.jersey.containers-jersey-container-servlet-core-2.27.jar:?]
> at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) ~[org.glassfish.jersey.containers-jersey-container-servlet-core-2.27.jar:?]
> at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) ~[org.glassfish.jersey.containers-jersey-container-servlet-core-2.27.jar:?]
> at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) ~[org.glassfish.jersey.containers-jersey-container-servlet-core-2.27.jar:?]
> at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:763) ~[org.eclipse.jetty-jetty-servlet-9.4.29.v20200521.jar:9.4.29.v20200521]
> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1631) ~[org.eclipse.jetty-jetty-servlet-9.4.29.v20200521.jar:9.4.29.v20200521]
> at org.apache.pulsar.broker.web.ResponseHandlerFilter.doFilter(ResponseHandlerFilter.java:53) ~[org.apache.pulsar-pulsar-broker-2.6.1.jar:2.6.1]
> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1618) ~[org.eclipse.jetty-jetty-servlet-9.4.29.v20200521.jar:9.4.29.v20200521]
> at org.apache.pulsar.broker.web.AuthenticationFilter.doFilter(AuthenticationFilter.java:82) ~[org.apache.pulsar-pulsar-broker-common-2.6.1.jar:2.6.1]
> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1618) ~[org.eclipse.jetty-jetty-servlet-9.4.29.v20200521.jar:9.4.29.v20200521]
> at org.apache.pulsar.broker.intercept.BrokerInterceptor$BrokerInterceptorDisabled.onWebServiceRequest(BrokerInterceptor.java:73) ~[org.apache.pulsar-pulsar-broker-2.6.1.jar:2.6.1]
> at org.apache.pulsar.broker.web.EventListenerFilter.doFilter(EventListenerFilter.java:46) ~[org.apache.pulsar-pulsar-broker-2.6.1.jar:2.6.1]
> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1618) ~[org.eclipse.jetty-jetty-servlet-9.4.29.v20200521.jar:9.4.29.v20200521]
> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:549) ~[org.eclipse.jetty-jetty-servlet-9.4.29.v20200521.jar:9.4.29.v20200521]
> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[org.eclipse.jetty-jetty-server-9.4.29.v20200521.jar:9.4.29.v20200521]
> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1610) ~[org.eclipse.jetty-jetty-server-9.4.29.v20200521.jar:9.4.29.v20200521]
> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[org.eclipse.jetty-jetty-server-9.4.29.v20200521.jar:9.4.29.v20200521]
> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1369) ~[org.eclipse.jetty-jetty-server-9.4.29.v20200521.jar:9.4.29.v20200521]
> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) ~[org.eclipse.jetty-jetty-server-9.4.29.v20200521.jar:9.4.29.v20200521]
> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:489) ~[org.eclipse.jetty-jetty-servlet-9.4.29.v20200521.jar:9.4.29.v20200521]
> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1580) ~[org.eclipse.jetty-jetty-server-9.4.29.v20200521.jar:9.4.29.v20200521]
> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) ~[org.eclipse.jetty-jetty-server-9.4.29.v20200521.jar:9.4.29.v20200521]
> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1284) ~[org.eclipse.jetty-jetty-server-9.4.29.v20200521.jar:9.4.29.v20200521]
> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[org.eclipse.jetty-jetty-server-9.4.29.v20200521.jar:9.4.29.v20200521]
> at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:221) ~[org.eclipse.jetty-jetty-server-9.4.29.v20200521.jar:9.4.29.v20200521]
> at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146) ~[org.eclipse.jetty-jetty-server-9.4.29.v20200521.jar:9.4.29.v20200521]
> at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:173) ~[org.eclipse.jetty-jetty-server-9.4.29.v20200521.jar:9.4.29.v20200521]
> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[org.eclipse.jetty-jetty-server-9.4.29.v20200521.jar:9.4.29.v20200521]
> at org.eclipse.jetty.server.Server.handle(Server.java:501) ~[org.eclipse.jetty-jetty-server-9.4.29.v20200521.jar:9.4.29.v20200521]
> at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383) ~[org.eclipse.jetty-jetty-server-9.4.29.v20200521.jar:9.4.29.v20200521]
> at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556) [org.eclipse.jetty-jetty-server-9.4.29.v20200521.jar:9.4.29.v20200521]
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375) [org.eclipse.jetty-jetty-server-9.4.29.v20200521.jar:9.4.29.v20200521]
> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273) [org.eclipse.jetty-jetty-server-9.4.29.v20200521.jar:9.4.29.v20200521]
> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) [org.eclipse.jetty-jetty-io-9.4.29.v20200521.jar:9.4.29.v20200521]
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) [org.eclipse.jetty-jetty-io-9.4.29.v20200521.jar:9.4.29.v20200521]
> at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:543) [org.eclipse.jetty-jetty-io-9.4.29.v20200521.jar:9.4.29.v20200521]
> at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:398) [org.eclipse.jetty-jetty-io-9.4.29.v20200521.jar:9.4.29.v20200521]
> at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:161) [org.eclipse.jetty-jetty-io-9.4.29.v20200521.jar:9.4.29.v20200521]
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) [org.eclipse.jetty-jetty-io-9.4.29.v20200521.jar:9.4.29.v20200521]
> at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) [org.eclipse.jetty-jetty-io-9.4.29.v20200521.jar:9.4.29.v20200521]
> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) [org.eclipse.jetty-jetty-util-9.4.29.v20200521.jar:9.4.29.v20200521]
> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) [org.eclipse.jetty-jetty-util-9.4.29.v20200521.jar:9.4.29.v20200521]
> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) [org.eclipse.jetty-jetty-util-9.4.29.v20200521.jar:9.4.29.v20200521]
> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) [org.eclipse.jetty-jetty-util-9.4.29.v20200521.jar:9.4.29.v20200521]
> at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375) [org.eclipse.jetty-jetty-util-9.4.29.v20200521.jar:9.4.29.v20200521]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_252]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_252]
> at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.48.Final.jar:4.1.48.Final]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] devinbost edited a comment on issue #7911: Unable to produce to specific topic in /public/default after upgrading to 2.6.1
Posted by GitBox <gi...@apache.org>.
devinbost edited a comment on issue #7911:
URL: https://github.com/apache/pulsar/issues/7911#issuecomment-681157405
When I try to lookup the problematic topic, I get a timeout as well:
```
root@server26:/pulsar# bin/pulsar-admin topics lookup persistent://public/default/canary
null
Reason: java.util.concurrent.TimeoutException
```
However, if I lookup topics in that same namespace that don't actually exist, instead of getting timeouts, I get actual servers in the cluster.
```
root@server26:/pulsar# bin/pulsar-admin topics lookup persistent://public/default/canary89000
"pulsar+ssl://server25.mydomain.com:6651"
root@server26:/pulsar# bin/pulsar-admin topics lookup persistent://public/default/canary8900
"pulsar+ssl://server28.mydomain.com:6651"
root@server26:/pulsar# bin/pulsar-admin topics lookup persistent://public/default/canary890
"pulsar+ssl://server26.mydomain.com:6651"
root@server26:/pulsar# bin/pulsar-admin topics lookup persistent://public/default/canary89
"pulsar+ssl://server28.mydomain.com:6651"
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] devinbost commented on issue #7911: Unable to produce to specific topic in /public/default after upgrading to 2.6.1
Posted by GitBox <gi...@apache.org>.
devinbost commented on issue #7911:
URL: https://github.com/apache/pulsar/issues/7911#issuecomment-681191487
@sijie or @codelipenghui maybe one of you will have ideas.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] devinbost commented on issue #7911: Unable to produce to specific topic in /public/default after upgrading to 2.6.1
Posted by GitBox <gi...@apache.org>.
devinbost commented on issue #7911:
URL: https://github.com/apache/pulsar/issues/7911#issuecomment-681157405
When I try to lookup the problematic topic, I get a timeout as well:
```
root@server26:/pulsar# bin/pulsar-admin topics lookup persistent://public/default/canary
null
Reason: java.util.concurrent.TimeoutException
```
However, if I lookup topics in that same namespace that don't actually exist, instead of getting timeouts, I get actual servers in the cluster.
```
root@server26:/pulsar# bin/pulsar-admin topics lookup persistent://public/default/canary89000
"pulsar+ssl://server25.mydomain.com:6651"
root@server26:/pulsar# bin/pulsar-admin topics lookup persistent://public/default/canary8900
"pulsar+ssl://server28.mydomain.com:6651"
root@server26:/pulsar# bin/pulsar-admin topics lookup persistent://public/default/canary8900
"pulsar+ssl://server28.mydomain.com:6651"
root@server26:/pulsar# bin/pulsar-admin topics lookup persistent://public/default/canary890
"pulsar+ssl://server26.mydomain.com:6651"
root@server26:/pulsar# bin/pulsar-admin topics lookup persistent://public/default/canary89
"pulsar+ssl://server28.mydomain.com:6651"
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] devinbost commented on issue #7911: Unable to produce to specific topic in /public/default after upgrading to 2.6.1
Posted by GitBox <gi...@apache.org>.
devinbost commented on issue #7911:
URL: https://github.com/apache/pulsar/issues/7911#issuecomment-681190880
Apparently, I have seen those Zookeeper warnings before... They were mentioned with this issue: https://github.com/apache/pulsar/issues/5311
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org