You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/12/27 09:13:01 UTC
[GitHub] [pulsar] keyboardbobo opened a new issue #13523: kill the broker or ack takes a long time, which will cause the cluster traffic to drop instantaneously
keyboardbobo opened a new issue #13523:
URL: https://github.com/apache/pulsar/issues/13523
**Is your enhancement request related to a problem? Please describe.**
I have nine brokers，when I kill one, the traffic of the remaining eight brokers will drop within 30 seconds, reach a peak after 30 seconds, and then return to normal. I found that the sending thread is blocked:
`
private boolean canEnqueueRequest(SendCallback callback, long sequenceId, int payloadSize) {
try {
if (conf.isBlockIfQueueFull()) {
if (semaphore.isPresent()) {
semaphore.get().acquire();
}
..
`
In other words, if the Producer queue corresponding to some partitions is full, the sending thread will be blocked, affecting the sending of other partitions
**Describe the solution you'd like**
Isolate partitions to avoid sending each other
**Describe alternatives you've considered**
PartitionedProducerImpl can automatically exclude Producers whose queues are full
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin commented on issue #13523: kill the broker or ack takes a long time, which will cause the cluster traffic to drop instantaneously
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on issue #13523:
URL: https://github.com/apache/pulsar/issues/13523#issuecomment-1006328180
@keyboardbobo You can commit your source code if you want.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin commented on issue #13523: kill the broker or ack takes a long time, which will cause the cluster traffic to drop instantaneously
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on issue #13523:
URL: https://github.com/apache/pulsar/issues/13523#issuecomment-1005451135
Recently, I upgrade broker, also found this problem, when kill on broker, other broker's bundle traffic will drop off
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin commented on issue #13523: kill the broker or ack takes a long time, which will cause the cluster traffic to drop instantaneously
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on issue #13523:
URL: https://github.com/apache/pulsar/issues/13523#issuecomment-1002095851
Let me take a look at this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin edited a comment on issue #13523: kill the broker or ack takes a long time, which will cause the cluster traffic to drop instantaneously
Posted by GitBox <gi...@apache.org>.
gaozhangmin edited a comment on issue #13523:
URL: https://github.com/apache/pulsar/issues/13523#issuecomment-1004548610
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] keyboardbobo commented on issue #13523: kill the broker or ack takes a long time, which will cause the cluster traffic to drop instantaneously
Posted by GitBox <gi...@apache.org>.
keyboardbobo commented on issue #13523:
URL: https://github.com/apache/pulsar/issues/13523#issuecomment-1006314674
I modified the source code, and by isolating the problematic producer, I can successfully prevent the cluster traffic from dropping.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] keyboardbobo removed a comment on issue #13523: kill the broker or ack takes a long time, which will cause the cluster traffic to drop instantaneously
Posted by GitBox <gi...@apache.org>.
keyboardbobo removed a comment on issue #13523:
URL: https://github.com/apache/pulsar/issues/13523#issuecomment-1006314275
By isolating the problematic producer, I can prevent the cluster traffic from dropping
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] hangc0276 commented on issue #13523: kill the broker or ack takes a long time, which will cause the cluster traffic to drop instantaneously
Posted by GitBox <gi...@apache.org>.
hangc0276 commented on issue #13523:
URL: https://github.com/apache/pulsar/issues/13523#issuecomment-1002658450
It's an interesting issue. I will keep track it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on issue #13523: kill the broker or ack takes a long time, which will cause the cluster traffic to drop instantaneously
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #13523:
URL: https://github.com/apache/pulsar/issues/13523#issuecomment-1051438003
The issue had no activity for 30 days, mark with Stale label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] keyboardbobo commented on issue #13523: kill the broker or ack takes a long time, which will cause the cluster traffic to drop instantaneously
Posted by GitBox <gi...@apache.org>.
keyboardbobo commented on issue #13523:
URL: https://github.com/apache/pulsar/issues/13523#issuecomment-1006226499
Theoretically speaking, only a slow response from any broker or a slow network will drag down other producers, because the slowest producer will always be the bottleneck
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] keyboardbobo commented on issue #13523: kill the broker or ack takes a long time, which will cause the cluster traffic to drop instantaneously
Posted by GitBox <gi...@apache.org>.
keyboardbobo commented on issue #13523:
URL: https://github.com/apache/pulsar/issues/13523#issuecomment-1004558859
```
ProducerBuilder builder = client.newProducer(Schema.STRING)
.topic("persistent://...")
.sendTimeout(120, TimeUnit.SECONDS)
.compressionType(CompressionType.LZ4)
.enableBatching(true)
.batchingMaxPublishDelay(10, TimeUnit.MILLISECONDS)
.batchingMaxMessages(1000)
.blockIfQueueFull(true);
```
@gaozhangmin The client configuration is as above, and the memoryLimitBytes parameter is not set. The main problem now is that a single broker has a problem, which will cause the traffic of other brokers to also drop. It is not that I want to limit the client memory or queue size.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] keyboardbobo commented on issue #13523: kill the broker or ack takes a long time, which will cause the cluster traffic to drop instantaneously
Posted by GitBox <gi...@apache.org>.
keyboardbobo commented on issue #13523:
URL: https://github.com/apache/pulsar/issues/13523#issuecomment-1006314275
By isolating the problematic producer, I can prevent the cluster traffic from dropping
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin commented on issue #13523: kill the broker or ack takes a long time, which will cause the cluster traffic to drop instantaneously
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on issue #13523:
URL: https://github.com/apache/pulsar/issues/13523#issuecomment-1004548610
@hangc0276 @keyboardbobo This issue is thrown https://github.com/apache/pulsar/wiki/PIP-74%3A-Pulsar-client-memory-limits. It tried to control the whole client memory usage. not per partition. I think it's the expected behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org