You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/05/05 08:40:13 UTC
[GitHub] [pulsar] Shawyeok opened a new issue, #15445: CPU usage 100% when use Key_Shared
Shawyeok opened a new issue, #15445:
URL: https://github.com/apache/pulsar/issues/15445
**Describe the bug**
When use `Key_Shared`, if one consumer consume slowly and others are normal, `MessageRedeliveryController#messagesToRedeliver` will keep growing and could be very large.
`MessageRedeliveryController#getMessagesToReplayNow` will be very expensive cause `messagesToRedeliver.items()` is a O(n) operation in both time and space complexity.
https://github.com/apache/pulsar/blob/2b2e0c50183c71e954f5c6e8bfcd7e36130279a5/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/MessageRedeliveryController.java#L109-L122
In this situation, one thread cpu usage could be 100%, this could effects the publish latency of other topics on the same broker instance.
**To Reproduce**
Steps to reproduce the behavior:
1. start a standalone cluster
2. start a producer
```shell
$ bin/pulsar-perf produce -mk random persistent://public/default/test
```
4. start a consumer that consume very slowly and a normal consumer
```shell
$ bin/pulsar-perf consume -st Key_Shared persistent://public/default/test
$ bin/pulsar-perf consume -st Key_Shared -r 0.1 persistent://public/default/test
```
**Screenshots**
CPU flame graph
**Additional context**
- pulsar: `2.8.1`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #15445: CPU usage 100% when use Key_Shared
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on issue #15445:
URL: https://github.com/apache/pulsar/issues/15445#issuecomment-1336950793
The issue should be fixed by https://github.com/apache/pulsar/pull/17804
@zhanghaou @Shawyeok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] surahman commented on issue #15445: CPU usage 100% when use Key_Shared
Posted by GitBox <gi...@apache.org>.
surahman commented on issue #15445:
URL: https://github.com/apache/pulsar/issues/15445#issuecomment-1124175742
This is a very interesting issue to examine.
I think the time bound is likely `O(n*log(n))` due to the `TreeSet` generation (sorting) here:
https://github.com/apache/pulsar/blob/2b2e0c50183c71e954f5c6e8bfcd7e36130279a5/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/MessageRedeliveryController.java#L116
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] zhanghaou commented on issue #15445: CPU usage 100% when use Key_Shared
Posted by GitBox <gi...@apache.org>.
zhanghaou commented on issue #15445:
URL: https://github.com/apache/pulsar/issues/15445#issuecomment-1336947666
We also encounter such problem，topics in the same broker publish latency is very high. Any suggestions？ @merlimat @codelipenghui
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] zhanghaou commented on issue #15445: CPU usage 100% when use Key_Shared
Posted by GitBox <gi...@apache.org>.
zhanghaou commented on issue #15445:
URL: https://github.com/apache/pulsar/issues/15445#issuecomment-1336956654
@codelipenghui Does the fix pr can be cherry pick to branch 2.8.X?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] merlimat commented on issue #15445: CPU usage 100% when use Key_Shared
Posted by GitBox <gi...@apache.org>.
merlimat commented on issue #15445:
URL: https://github.com/apache/pulsar/issues/15445#issuecomment-1118911309
@Shawyeok Can you attach the flame graph directly?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui closed issue #15445: CPU usage 100% when use Key_Shared
Posted by GitBox <gi...@apache.org>.
codelipenghui closed issue #15445: CPU usage 100% when use Key_Shared
URL: https://github.com/apache/pulsar/issues/15445
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #15445: CPU usage 100% when use Key_Shared
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on issue #15445:
URL: https://github.com/apache/pulsar/issues/15445#issuecomment-1337176787
@zhanghaou We are only applying security patches for 2.8.x release for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Shawyeok commented on issue #15445: CPU usage 100% when use Key_Shared
Posted by GitBox <gi...@apache.org>.
Shawyeok commented on issue #15445:
URL: https://github.com/apache/pulsar/issues/15445#issuecomment-1119194141
> @Shawyeok Can you attach the flame graph directly?
@merlimat [cpu_profile.html.zip](https://github.com/apache/pulsar/files/8637078/cpu_profile.html.zip) attached
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Shawyeok commented on issue #15445: CPU usage 100% when use Key_Shared
Posted by GitBox <gi...@apache.org>.
Shawyeok commented on issue #15445:
URL: https://github.com/apache/pulsar/issues/15445#issuecomment-1118334156
> When use Key_Shared, if one consumer consume slowly and others are normal, MessageRedeliveryController#messagesToRedeliver will keep growing and could be very large.
Additional context:
The slow consumer will ends with zero availablePermits, and message positions will be recorded in `MessageRedeliveryController#messagesToRedeliver`, when none of the whole batch of messages sent to consumer, `isDispatcherStuckOnReplays` will be marked.
https://github.com/apache/pulsar/blob/2b2e0c50183c71e954f5c6e8bfcd7e36130279a5/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentStickyKeyDispatcherMultipleConsumers.java#L310-L330
Next time call `getMessagesToReplayNow` in readMoreEntries, it'll returns empty, then it'll read next batch of messages by cursor, therefor more message positions will be recorded.
https://github.com/apache/pulsar/blob/2b2e0c50183c71e954f5c6e8bfcd7e36130279a5/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentStickyKeyDispatcherMultipleConsumers.java#L436-L446
https://github.com/apache/pulsar/blob/2b2e0c50183c71e954f5c6e8bfcd7e36130279a5/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentStickyKeyDispatcherMultipleConsumers.java#L310-L330
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] WJL3333 commented on issue #15445: CPU usage 100% when use Key_Shared
Posted by GitBox <gi...@apache.org>.
WJL3333 commented on issue #15445:
URL: https://github.com/apache/pulsar/issues/15445#issuecomment-1279980754
fix by #17804
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on issue #15445: CPU usage 100% when use Key_Shared
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #15445:
URL: https://github.com/apache/pulsar/issues/15445#issuecomment-1153047958
The issue had no activity for 30 days, mark with Stale label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] zhanghaou commented on issue #15445: CPU usage 100% when use Key_Shared
Posted by GitBox <gi...@apache.org>.
zhanghaou commented on issue #15445:
URL: https://github.com/apache/pulsar/issues/15445#issuecomment-1337206490
@codelipenghui OK, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org