You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "luky116 (via GitHub)" <gi...@apache.org> on 2024/07/07 10:58:29 UTC
[I] [Bug] bin/pulsar-perf will cause the pulsar service to freeze [pulsar]
luky116 opened a new issue, #23007:
URL: https://github.com/apache/pulsar/issues/23007
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Read release policy
- [X] I understand that unsupported versions don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.
### Version
OS：Linux 4.19.91-24.8.an8.x86_64 #1 SMP Tue Aug 31 11:30:53 CST 2021 x86_64 x86_64 x86_64 GNU/Linux
Java：java version "17.0.11" 2024-04-16 LTS
Pulsar：v3.0.5
### Minimal reproduce step
1. start pulsar cluster
2. create a persistent topic
3. use bin/pulsar-perf produce tool to produce some produce message
4. Lookup request timeout, as below:
```
java.util.concurrent.ExecutionException: org.apache.pulsar.client.api.PulsarClientException$TimeoutException: Lookup request timeout {'durationMs': '30000', 'reqId':'3305977965004565187', 'remote':'xx.xx.xx.xx/10.224.129.240:6650', 'local':'/10.106.142.241:64755'}
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) ~[?:?]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) ~[?:?]
at org.apache.pulsar.testclient.PerformanceProducer.runProducer(PerformanceProducer.java:598) ~[org.apache.pulsar-pulsar-testclient-3.0.5.jar:3.0.5]
at org.apache.pulsar.testclient.PerformanceProducer.lambda$main$1(PerformanceProducer.java:399) ~[org.apache.pulsar-pulsar-testclient-3.0.5.jar:3.0.5]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.108.Final.jar:4.1.108.Final]
at java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: org.apache.pulsar.client.api.PulsarClientException$TimeoutException: Lookup request timeout {'durationMs': '30000', 'reqId':'3305977965004565187', 'remote':'xx.xx.xx.xx/10.224.129.240:6650', 'local':'/10.106.142.241:64755'}
at org.apache.pulsar.client.impl.ClientCnx.checkRequestTimeout(ClientCnx.java:1355) ~[org.apache.pulsar-pulsar-client-original-3.0.5.jar:3.0.5]
at org.apache.pulsar.common.util.Runnables$CatchingAndLoggingRunnable.run(Runnables.java:54) ~[org.apache.pulsar-pulsar-common-3.0.5.jar:3.0.5]
at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) ~[io.netty-netty-common-4.1.108.Final.jar:4.1.108.Final]
at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:159) ~[io.netty-netty-common-4.1.108.Final.jar:4.1.108.Final]
at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) ~[io.netty-netty-common-4.1.108.Final.jar:4.1.108.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) ~[io.netty-netty-common-4.1.108.Final.jar:4.1.108.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[io.netty-netty-common-4.1.108.Final.jar:4.1.108.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569) ~[io.netty-netty-transport-4.1.108.Final.jar:4.1.108.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[io.netty-netty-common-4.1.108.Final.jar:4.1.108.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[io.netty-netty-common-4.1.108.Final.jar:4.1.108.Final]
... 2 more
```
### What did you expect to see?
pulsar can produce and consume messages normally
### What did you see instead?
Pulsar is stuck and times out when producing messages
### Anything else?
_No response_
### Are you willing to submit a PR?
- [X] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] bin/pulsar-perf will cause the pulsar service to freeze [pulsar]
Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat commented on issue #23007:
URL: https://github.com/apache/pulsar/issues/23007#issuecomment-6380997686
Thanks for the report, @luky116. The issue only shows a client-side `Lookup request timeout`, which doesn't tell us what got stuck on the broker, and 3.0.x is no longer supported. Could you try to reproduce on a currently supported release (4.0.x, 4.2.x or 5.0.x)? If it still happens, please share the broker logs from the time of the timeout, a thread dump of the broker owning the topic (`jstack`), `pulsar-admin topics stats` / `stats-internal` for the topic, the load manager in use, and the exact pulsar-perf command. If there's no update, this issue will be closed in a few weeks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org