You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2023/01/10 08:43:15 UTC
[GitHub] [pulsar] TakaHiR07 opened a new issue, #19169: [Bug][txn] exclusive subscription would lead to high cpu usage when do tailing-read
TakaHiR07 opened a new issue, #19169:
URL: https://github.com/apache/pulsar/issues/19169
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Version
server and client version: pulsar-2.9.4
### Minimal reproduce step
do txn produce and txn consume on a 200-partition topic by pulsar-perf. The throughput is 10MB/s, batchSize is 10, subscriptionType is exclusive. It is a tailing read, consuming the latest message
produce config is :`-txn -nmt 1000 -time 0 -s 1024 -i 60 -bm 10 -b 1000 -bb 4194304 -r 10000 -mk random -threads 3`; consume config is : `-txn -nmt 2000 -time 0 -i 60 -s sub_test_txn_p200 -ss sub_test_txn_p200 -sp Latest -ioThreads 1 -n 1`
### What did you expect to see?
The cpu usage should not be so high when it is exclusive subscription, tailing read and txn consume.
Actually the tailing read should not consume more resource than catchup read.
### What did you see instead?
1. when it is tailing read and exclusive subscription and txn consume, the cpu usage is 80%. But when it turn to catchup read, the cpu usage is 45%.
2. Throughput would not influence the cpu usage, but if topic have more partition, the cpu usage is higher.
I add some log in code and have found the cpu is consumed in "broker-topic-workers", which is mostly do PersistentDispatcherSingleActiveConsumer#readEntriesComplete and PersistentDispatcherSingleActiveConsumer#dispatchEntriesToConsumer.
I also add a metric of thread pool "broker-topic-workers", and found that when do catchup read
2. Throughput would not influence the cpu usage, but if topic have more partition, the cpu usage is higher.
3. I add some log in code and have found the cpu is consumed in "broker-topic-workers", which is mostly do PersistentDispatcherSingleActiveConsumer#readEntriesComplete and PersistentDispatcherSingleActiveConsumer#dispatchEntriesToConsumer.
4. I also add a metric of thread pool "broker-topic-workers", and found that when do catchup read. the completed task in thread pool is total 150K * 8 thread = 1200K/s, while the read throughput is 10K message/s. There are too many task in this thread pool.
### Anything else?
Although tested in 2.9.4, I guess this problem may also occur in master branch.
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] TakaHiR07 commented on issue #19169: [Bug][txn] exclusive subscription would lead to high cpu usage when do tailing-read
Posted by GitBox <gi...@apache.org>.
TakaHiR07 commented on issue #19169:
URL: https://github.com/apache/pulsar/issues/19169#issuecomment-1379746393
> @TakaHiR07 #14286 has not been fixed. this problem is the same as #14286
Thanks for your view. I guess this high cpu problem is easily reproduced and need to be fixed in the next pulsar version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug][txn] exclusive subscription would lead to high cpu usage when do tailing-read [pulsar]
Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat closed issue #19169: [Bug][txn] exclusive subscription would lead to high cpu usage when do tailing-read
URL: https://github.com/apache/pulsar/issues/19169
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] congbobo184 commented on issue #19169: [Bug][txn] exclusive subscription would lead to high cpu usage when do tailing-read
Posted by GitBox <gi...@apache.org>.
congbobo184 commented on issue #19169:
URL: https://github.com/apache/pulsar/issues/19169#issuecomment-1383208080
oh, sure. I will make https://github.com/apache/pulsar/pull/14286 merge quickly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug][txn] exclusive subscription would lead to high cpu usage when do tailing-read [pulsar]
Posted by "TakaHiR07 (via GitHub)" <gi...@apache.org>.
TakaHiR07 commented on issue #19169:
URL: https://github.com/apache/pulsar/issues/19169#issuecomment-2165031292
> oh, sure. I will make #14286 merge quickly
@congbobo184 Is you willing to continue working on this pr ? I test transaction in pulsar-3.0 and issue still exist. And I want to cherry-pick this pr to pulsar-3.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug][txn] exclusive subscription would lead to high cpu usage when do tailing-read [pulsar]
Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat commented on issue #19169:
URL: https://github.com/apache/pulsar/issues/19169#issuecomment-6386668120
Thanks @TakaHiR07 for the report. This was reported on Pulsar 2.9, which is no longer supported; the supported releases are 4.0.x, 4.2.x and 5.0.x. Much of the code involved has changed since, and the problem hasn't been confirmed on a current release, so I'm closing it as part of cleaning up old reports. If you still see it on a supported release, please open a new issue with the version and steps to reproduce, and link this one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug][txn] exclusive subscription would lead to high cpu usage when do tailing-read [pulsar]
Posted by "dao-jun (via GitHub)" <gi...@apache.org>.
dao-jun commented on issue #19169:
URL: https://github.com/apache/pulsar/issues/19169#issuecomment-2119269621
it looks the issue fixed by https://github.com/apache/pulsar/pull/14667, closing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on issue #19169: [Bug][txn] exclusive subscription would lead to high cpu usage when do tailing-read
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #19169:
URL: https://github.com/apache/pulsar/issues/19169#issuecomment-1430647965
The issue had no activity for 30 days, mark with Stale label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug][txn] exclusive subscription would lead to high cpu usage when do tailing-read [pulsar]
Posted by "dao-jun (via GitHub)" <gi...@apache.org>.
dao-jun closed issue #19169: [Bug][txn] exclusive subscription would lead to high cpu usage when do tailing-read
URL: https://github.com/apache/pulsar/issues/19169
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] TakaHiR07 commented on issue #19169: [Bug][txn] exclusive subscription would lead to high cpu usage when do tailing-read
Posted by GitBox <gi...@apache.org>.
TakaHiR07 commented on issue #19169:
URL: https://github.com/apache/pulsar/issues/19169#issuecomment-1378480776
@congbobo184 Can you help take a look of this problem ? Is this problem the same as https://github.com/apache/pulsar/pull/14286 described, and has not been fix ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] congbobo184 commented on issue #19169: [Bug][txn] exclusive subscription would lead to high cpu usage when do tailing-read
Posted by GitBox <gi...@apache.org>.
congbobo184 commented on issue #19169:
URL: https://github.com/apache/pulsar/issues/19169#issuecomment-1379696469
@TakaHiR07 https://github.com/apache/pulsar/pull/14286 has not been fixed. this problem is the same as https://github.com/apache/pulsar/pull/14286
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org