You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2019/01/10 01:01:52 UTC
[GitHub] rdhabalia opened a new pull request #3347: Support Add-entry
timeout at broker to avoid stuck topics
rdhabalia opened a new pull request #3347: Support Add-entry timeout at broker to avoid stuck topics
URL: https://github.com/apache/pulsar/pull/3347
### Motivation
Recently and in past, we have seen few instances where bookie crashes when it goes out of memory and kernel panics and broker doesn't complete add-entry callback because of that topics get stuck on pending-writes.
```
2010-01-01T00:18:19+0000 2010-01-01 00:18:19,594 21907 report_stuck:556 WARNING persistent://prop1/global/prod-rt-mxt/extractions-partition-0 is stuck (broker1.us-west1.com): waiting for write
2010-01-01T00:18:22+0000 2010-01-01 00:18:22,654 21907 report_stuck:556 WARNING persistent://prop1/global/mbr-events/27_bf1 is stuck (broker1.us-west1.com): waiting for write
2010-01-01T00:19:10+0000 2010-01-01 00:19:10,790 21907 report_stuck:556 WARNING persistent://prop1/global/mbr-events/30_bf1 is stuck (broker1.us-west1.com): waiting for write
2010-01-01T00:19:14+0000 2010-01-01 00:19:14,692 21907 report_stuck:556 WARNING persistent://prop1/global/mbr-events/35_gq2 is stuck (broker1.us-west1.com): waiting for write
2010-01-01T00:19:15+0000 2010-01-01 00:19:15,809 21907 report_stuck:556 WARNING persistent://prop1/global/jedi-events/batchevents-partition-35 is stuck (broker1.us-west1.com): waiting for write
2010-01-01T00:19:25+0000 2010-01-01 00:19:25,127 21907 report_stuck:556 WARNING persistent://prop1/global/jedi-events/mailevents-partition-24 is stuck (broker1.us-west1.com): waiting for write
```
internal-stats
```
{
"entriesAddedCounter" : 66066317,
"numberOfEntries" : 14129359,
"totalSize" : 145505085493,
"currentLedgerEntries" : 238938,
"currentLedgerSize" : 3715327400,
"lastLedgerCreatedTimestamp" : "2018-11-27 00:03:03.125+0000",
"waitingCursorsCount" : 0,
"pendingAddEntriesCount" : 252514,
"lastConfirmedEntry" : "489781963:26121",
"state" : "ClosingLedger",
"ledgers" : [ {
"ledgerId" : 489765238,
"entries" : 559855,
"size" : 6928710817
}, {
...
```
### Modifications
Add support of write timeout which can be disable by configuring timeout=0.
### Result
topic will not stuck in pending write in case add-entry callback doesn't completes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services