You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/07/15 10:52:45 UTC
[GitHub] [pulsar] KyrieXu11 opened a new issue, #16625: some consumers of several topics can't consume normally
KyrieXu11 opened a new issue, #16625:
URL: https://github.com/apache/pulsar/issues/16625
**System Info**
Pulsar Version: 2.10.0
Zk version: zookeeper in pulsar2.9.0
OS: Linux x64
Hardware: 5 Brokers and 5 Bookies hybrid deployment in a machine. 32C128G(4T*12) * 5
3 zookeeper nodes 4C32G200GB * 3
Broker Conf File: [[broker.conf](https://github.com/apache/pulsar/files/9119526/broker.txt)](https://github.com/apache/pulsar/files/9119526/broker.txt)
Bookie Conf File: [[bookkeeper.conf](https://github.com/apache/pulsar/files/9119539/bookkeeper.txt)](https://github.com/apache/pulsar/files/9119539/bookkeeper.txt)
**Describe the bug**
After initialized Pulsar cluster,(doc of deployment: https://pulsar.apache.org/docs/2.10.0/deploy-bare-metal/). we synchronized all topics from Pulsar 2.6.0 to Pulsar 2.10.0.After that we started out producer and consumer program,copy all flow from pulsar 2.6.0 to pulsar 2.10.0.But,after a period of time,we found that some consumers of some topics will not consume normally.The screenshot below illustrates this situation.
there are some error logs in our consumer program
```
time="2022-07-15T13:27:49+08:00" level=error msg="[Discarding corrupted message]" consumerID=4078 msgID="ledgerId:60580 entryId:5516 partition:-1 " name=fwtzx subscription=xxxxx topic="persistent://xxxxx-xxx/xxx/56f085cd0e5b40b687a80a674c189229" validationError=ChecksumMismatch
time="2022-07-15T13:27:49+08:00" level=error msg="[handle message Id: ledgerId:60580 entryId:5516 partition:-1 ]" consumerID=4078 error="checksum mismatch received: 0x1a8d6a22 computed: 0xc3262786" local_addr="x.x.x.x:59836" remote_addr="pulsar://x.x.x.x:6650"
```
there are also some error logs in broker log at the same time.
```
2022-07-15T13:27:18,853+0800 [pulsar-io-4-62] ERROR org.apache.pulsar.broker.service.Consumer - [PersistentSubscription{topic=persistent://xxxx/xxxx/56f085cd0e5b40b687a80a674c189229, name=}] [4078] Received ack for corrupted message at 60580:5440 - Reason: ChecksumMismatch
2022-07-15T13:27:49,730+0800 [pulsar-io-4-62] ERROR org.apache.pulsar.broker.service.Consumer - [PersistentSubscription{topic=persistent://xxx/xxxx/56f085cd0e5b40b687a80a674c189229, name=}] [4078] Received ack for corrupted message at 60580:5516 - Reason: ChecksumMismatch
```
**To Reproduce**
it's a pity that we can't give our producer and consumer program,but we can confirm that our program will not definitely affect this situation,because it works correctly with pulsar 2.6.0.
1. start pulsar 2.10.0 cluster.
2. start our producer and consumer program.
3. wait for a period time, use pulsar admin stats command or pulsar-manager to get stats,so you can find that consumer out rate is zero.
**Expected behavior**
All of consumers of all of topics can consume normally.
**Screenshots**
in the screenshot above,`demzi`is abnormal consumer，but in the screenshot below that the comsumer named `demzi` belong to the different topic can consume normally.
in the screenshot below that the comsumer named `demzi` belong to the different topic in screenshot above can't consume normally
topics in all of screenshots above are belong to the same namespace and tenant.
json info below is stats and internal stats of topic named`ed797eee19b14f1db9048194a3853ae7`.
`stats info`
```
{
"msgRateIn" : 5.31688308586027,
"msgThroughputIn" : 1739070.470715847,
"msgRateOut" : 10.63376598919374,
"msgThroughputOut" : 3478140.881904425,
"bytesInCounter" : 23288660172,
"msgInCounter" : 79811,
"bytesOutCounter" : 45274531315,
"msgOutCounter" : 154473,
"averageMsgSize" : 327084.5799375831,
"msgChunkPublished" : false,
"storageSize" : 280783674607,
"backlogSize" : 0,
"publishRateLimitedTimes" : 0,
"earliestMsgPublishTimeInBacklogs" : 0,
"offloadedStorageSize" : 0,
"lastOffloadLedgerId" : 0,
"lastOffloadSuccessTimeStamp" : 0,
"lastOffloadFailureTimeStamp" : 0,
"publishers" : [ {
"accessMode" : "Shared",
"msgRateIn" : 2.7334445964188916,
"msgThroughputIn" : 897628.3040433243,
"averageMsgSize" : 328387.23170731706,
"chunkedMessageRate" : 0.0,
"producerId" : 4162,
"supportsPartialProducer" : false,
"metadata" : { },
"connectedSince" : "2022-07-15T11:33:40.965+08:00",
"producerName" : "1296"
}, {
"accessMode" : "Shared",
"msgRateIn" : 2.583438489441378,
"msgThroughputIn" : 841442.1666725228,
"averageMsgSize" : 325706.29032258067,
"chunkedMessageRate" : 0.0,
"producerId" : 4162,
"supportsPartialProducer" : false,
"metadata" : { },
"connectedSince" : "2022-07-15T11:30:47.942+08:00",
"producerName" : "646"
} ],
"waitingPublishers" : 0,
"subscriptions" : {
"perf-test" : {
"msgRateOut" : 5.316882956521688,
"msgThroughputOut" : 1739070.4283761359,
"bytesOutCounter" : 21900023316,
"msgOutCounter" : 74335,
"msgRateRedeliver" : 0.0,
"chunkedMessageRate" : 0,
"msgBacklog" : 0,
"backlogSize" : 0,
"earliestMsgPublishTimeInBacklog" : 0,
"msgBacklogNoDelayed" : 0,
"blockedSubscriptionOnUnackedMsgs" : false,
"msgDelayed" : 0,
"unackedMessages" : 0,
"type" : "Shared",
"msgRateExpired" : 0.0,
"totalMsgExpired" : 0,
"lastExpireTimestamp" : 0,
"lastConsumedFlowTimestamp" : 1657870779250,
"lastConsumedTimestamp" : 1657870930521,
"lastAckedTimestamp" : 1657870930526,
"lastMarkDeleteAdvancedTimestamp" : 1657870930526,
"consumers" : [ {
"msgRateOut" : 2.7001098325026622,
"msgThroughputOut" : 889123.5669237879,
"bytesOutCounter" : 18955307813,
"msgOutCounter" : 65434,
"msgRateRedeliver" : 0.0,
"chunkedMessageRate" : 0.0,
"consumerName" : "wvhxo",
"availablePermits" : 566,
"unackedMessages" : 0,
"avgMessagesPerEntry" : 1,
"blockedConsumerOnUnackedMsgs" : false,
"lastAckedTimestamp" : 1657870930526,
"lastConsumedTimestamp" : 1657870930521,
"metadata" : { },
"connectedSince" : "2022-07-15T11:49:26.338+08:00"
}, {
"msgRateOut" : 2.6167731240190264,
"msgThroughputOut" : 849946.861452348,
"bytesOutCounter" : 2944715503,
"msgOutCounter" : 8901,
"msgRateRedeliver" : 0.0,
"chunkedMessageRate" : 0.0,
"consumerName" : "eiwpn",
"availablePermits" : 599,
"unackedMessages" : 0,
"avgMessagesPerEntry" : 1,
"blockedConsumerOnUnackedMsgs" : false,
"lastAckedTimestamp" : 1657870930432,
"lastConsumedTimestamp" : 1657870930430,
"metadata" : { },
"connectedSince" : "2022-07-15T14:46:36.316+08:00"
} ],
"isDurable" : true,
"isReplicated" : false,
"allowOutOfOrderDelivery" : false,
"consumersAfterMarkDeletePosition" : { },
"nonContiguousDeletedMessagesRanges" : 0,
"nonContiguousDeletedMessagesRangesSerializedSize" : 18,
"subscriptionProperties" : { },
"replicated" : false,
"durable" : true
},
"subscription_name" : {
"msgRateOut" : 5.316883032672052,
"msgThroughputOut" : 1739070.453528289,
"bytesOutCounter" : 23374507999,
"msgOutCounter" : 80138,
"msgRateRedeliver" : 0.0,
"chunkedMessageRate" : 0,
"msgBacklog" : 0,
"backlogSize" : 0,
"earliestMsgPublishTimeInBacklog" : 0,
"msgBacklogNoDelayed" : 0,
"blockedSubscriptionOnUnackedMsgs" : false,
"msgDelayed" : 0,
"unackedMessages" : 0,
"type" : "Shared",
"msgRateExpired" : 0.0,
"totalMsgExpired" : 0,
"lastExpireTimestamp" : 1657865239011,
"lastConsumedFlowTimestamp" : 1657870930144,
"lastConsumedTimestamp" : 1657870930521,
"lastAckedTimestamp" : 1657870930535,
"lastMarkDeleteAdvancedTimestamp" : 1657870930535,
"consumers" : [ {
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"bytesOutCounter" : 12470952333,
"msgOutCounter" : 46230,
"msgRateRedeliver" : 0.0,
"chunkedMessageRate" : 0.0,
"consumerName" : "demzi",
"availablePermits" : 0,
"unackedMessages" : 0,
"avgMessagesPerEntry" : 1,
"blockedConsumerOnUnackedMsgs" : false,
"lastAckedTimestamp" : 1657864566575,
"lastConsumedTimestamp" : 1657864566531,
"metadata" : { },
"connectedSince" : "2022-07-15T11:31:11.667+08:00"
}, {
"msgRateOut" : 2.6501078628351578,
"msgThroughputOut" : 881596.532124933,
"bytesOutCounter" : 6324041349,
"msgOutCounter" : 19852,
"msgRateRedeliver" : 0.0,
"chunkedMessageRate" : 0.0,
"consumerName" : "ejynz",
"availablePermits" : 8,
"unackedMessages" : 0,
"avgMessagesPerEntry" : 1,
"blockedConsumerOnUnackedMsgs" : false,
"lastAckedTimestamp" : 1657870930438,
"lastConsumedTimestamp" : 1657870930430,
"metadata" : { },
"connectedSince" : "2022-07-15T14:07:39.859+08:00"
}, {
"msgRateOut" : 2.666775169836894,
"msgThroughputOut" : 857473.921403356,
"bytesOutCounter" : 4579514317,
"msgOutCounter" : 14056,
"msgRateRedeliver" : 0.0,
"chunkedMessageRate" : 0.0,
"consumerName" : "vsbvh",
"availablePermits" : 9,
"unackedMessages" : 0,
"avgMessagesPerEntry" : 1,
"blockedConsumerOnUnackedMsgs" : false,
"lastAckedTimestamp" : 1657870930535,
"lastConsumedTimestamp" : 1657870930521,
"metadata" : { },
"connectedSince" : "2022-07-15T14:14:05.845+08:00"
} ],
"isDurable" : true,
"isReplicated" : false,
"allowOutOfOrderDelivery" : false,
"consumersAfterMarkDeletePosition" : { },
"nonContiguousDeletedMessagesRanges" : 0,
"nonContiguousDeletedMessagesRangesSerializedSize" : 18,
"subscriptionProperties" : { },
"replicated" : false,
"durable" : true
}
},
"replication" : { },
"deduplicationStatus" : "Disabled",
"nonContiguousDeletedMessagesRanges" : 0,
"nonContiguousDeletedMessagesRangesSerializedSize" : 36,
"compaction" : {
"lastCompactionRemovedEventCount" : 0,
"lastCompactionSucceedTimestamp" : 0,
"lastCompactionFailedTimestamp" : 0,
"lastCompactionDurationTimeInMills" : 0
}
}
```
`internal-stats info`
```
{
"entriesAddedCounter" : 81741,
"numberOfEntries" : 1045352,
"totalSize" : 281430250165,
"currentLedgerEntries" : 933,
"currentLedgerSize" : 311482997,
"lastLedgerCreatedTimestamp" : "2022-07-15T15:45:17.915+08:00",
"waitingCursorsCount" : 2,
"pendingAddEntriesCount" : 0,
"lastConfirmedEntry" : "61763:932",
"state" : "LedgerOpened",
"ledgers" : [ {
"ledgerId" : 64,
"entries" : 4592,
"size" : 2147522620,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 8591,
"entries" : 4374,
"size" : 2147602651,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 8610,
"entries" : 4364,
"size" : 2147955554,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 8620,
"entries" : 4449,
"size" : 2147514383,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 8647,
"entries" : 4326,
"size" : 2147809213,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 8661,
"entries" : 4458,
"size" : 2147626310,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 8680,
"entries" : 4506,
"size" : 2147864017,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 8695,
"entries" : 4634,
"size" : 2147603004,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 8724,
"entries" : 5434,
"size" : 2147503578,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 8744,
"entries" : 6229,
"size" : 2147886295,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 9594,
"entries" : 7881,
"size" : 2147609865,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 9779,
"entries" : 11211,
"size" : 2147495663,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 9821,
"entries" : 27866,
"size" : 2147495679,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 10744,
"entries" : 16321,
"size" : 2147625044,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 10814,
"entries" : 9980,
"size" : 2147593524,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 10855,
"entries" : 9567,
"size" : 2147552339,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 11720,
"entries" : 8068,
"size" : 2147491135,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 11793,
"entries" : 7231,
"size" : 2147498641,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 11850,
"entries" : 890,
"size" : 281475581,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 14161,
"entries" : 84,
"size" : 26211504,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 18436,
"entries" : 8580,
"size" : 2147514842,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 21794,
"entries" : 7360,
"size" : 2147613104,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 21804,
"entries" : 7418,
"size" : 2147827304,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 21823,
"entries" : 7166,
"size" : 2147635029,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 22342,
"entries" : 6353,
"size" : 2147836935,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 22746,
"entries" : 6130,
"size" : 2147490889,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 23434,
"entries" : 6164,
"size" : 2147765542,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 23473,
"entries" : 6444,
"size" : 2147675767,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 23489,
"entries" : 6512,
"size" : 2147616659,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 23505,
"entries" : 6589,
"size" : 2147487039,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 23534,
"entries" : 7412,
"size" : 2147704633,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 23556,
"entries" : 7780,
"size" : 2147525348,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 23566,
"entries" : 7068,
"size" : 2147775271,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 23942,
"entries" : 6174,
"size" : 2147646176,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 24541,
"entries" : 5621,
"size" : 2147491980,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 24605,
"entries" : 5355,
"size" : 2147654188,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 24638,
"entries" : 5031,
"size" : 2147700934,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 24657,
"entries" : 4903,
"size" : 2147496553,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 24678,
"entries" : 4706,
"size" : 2147835265,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 24702,
"entries" : 4658,
"size" : 2147819049,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 24717,
"entries" : 4617,
"size" : 2147510933,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 24729,
"entries" : 4556,
"size" : 2147860016,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 24741,
"entries" : 4502,
"size" : 2147711708,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 24762,
"entries" : 4538,
"size" : 2147954765,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 24771,
"entries" : 4543,
"size" : 2147696369,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 24787,
"entries" : 4514,
"size" : 2147591010,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 24821,
"entries" : 4497,
"size" : 2147750756,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 24832,
"entries" : 4500,
"size" : 2147666706,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 24850,
"entries" : 4525,
"size" : 2147782415,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 24876,
"entries" : 4570,
"size" : 2147908568,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 24890,
"entries" : 4618,
"size" : 2147609937,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 25233,
"entries" : 4555,
"size" : 2147882063,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 25759,
"entries" : 4785,
"size" : 2147686484,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 25859,
"entries" : 5329,
"size" : 2147606636,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 25894,
"entries" : 5992,
"size" : 2147667546,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 25925,
"entries" : 6173,
"size" : 2147506275,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 25956,
"entries" : 7066,
"size" : 2147750143,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 25984,
"entries" : 8489,
"size" : 2147529113,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 26012,
"entries" : 14947,
"size" : 2147536584,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 26051,
"entries" : 31800,
"size" : 2147513278,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 26982,
"entries" : 36271,
"size" : 2147528297,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 27085,
"entries" : 19741,
"size" : 2147556293,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 27121,
"entries" : 14156,
"size" : 2147499263,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 27925,
"entries" : 11082,
"size" : 2147606053,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 28045,
"entries" : 10730,
"size" : 2147618502,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 28113,
"entries" : 9982,
"size" : 2147510571,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 28162,
"entries" : 8951,
"size" : 2147609396,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 28200,
"entries" : 8326,
"size" : 2147682990,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 28252,
"entries" : 7673,
"size" : 2147626037,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 28295,
"entries" : 7329,
"size" : 2147668143,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 28309,
"entries" : 7002,
"size" : 2147735127,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 28615,
"entries" : 8688,
"size" : 2147772030,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 29181,
"entries" : 8251,
"size" : 2147629319,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 29266,
"entries" : 7572,
"size" : 2147619347,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 29340,
"entries" : 6969,
"size" : 2147536513,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 29373,
"entries" : 6868,
"size" : 2147695876,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 29402,
"entries" : 7140,
"size" : 2147609479,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 29435,
"entries" : 7468,
"size" : 2147699877,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 29465,
"entries" : 7122,
"size" : 2147754206,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 29516,
"entries" : 6500,
"size" : 2147603141,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 29555,
"entries" : 6642,
"size" : 2147666946,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 29591,
"entries" : 6559,
"size" : 2147547869,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 30291,
"entries" : 6280,
"size" : 2147807260,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 30462,
"entries" : 6313,
"size" : 2147693212,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 30527,
"entries" : 6376,
"size" : 2147943639,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 30588,
"entries" : 6254,
"size" : 2147486942,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 30643,
"entries" : 5830,
"size" : 1998608293,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 31991,
"entries" : 4701,
"size" : 2147840057,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 36490,
"entries" : 4678,
"size" : 2147560079,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 36512,
"entries" : 4520,
"size" : 2147943263,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 36525,
"entries" : 4488,
"size" : 2147800229,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 36551,
"entries" : 4399,
"size" : 2147663049,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 36573,
"entries" : 4476,
"size" : 2191779099,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 36589,
"entries" : 4385,
"size" : 2147561734,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 36612,
"entries" : 4375,
"size" : 2147509480,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 36624,
"entries" : 4433,
"size" : 2147920410,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 36638,
"entries" : 4388,
"size" : 2147517038,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 36651,
"entries" : 4372,
"size" : 2147718228,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 36676,
"entries" : 4395,
"size" : 2147660681,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 36689,
"entries" : 4415,
"size" : 2147956505,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 36702,
"entries" : 4468,
"size" : 2147758796,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 36726,
"entries" : 4552,
"size" : 2147842691,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 36739,
"entries" : 4745,
"size" : 2147611369,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 36754,
"entries" : 5212,
"size" : 2147553417,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 36768,
"entries" : 5654,
"size" : 2147596502,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 36801,
"entries" : 6343,
"size" : 2147671753,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 36824,
"entries" : 6425,
"size" : 2147723556,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 37794,
"entries" : 7494,
"size" : 2147558261,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 37850,
"entries" : 8910,
"size" : 2147655113,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 37893,
"entries" : 12272,
"size" : 2147502031,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 37921,
"entries" : 32666,
"size" : 2147498467,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 37981,
"entries" : 37725,
"size" : 2147519078,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 39007,
"entries" : 20065,
"size" : 2147546779,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 39064,
"entries" : 14011,
"size" : 2147555342,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 39089,
"entries" : 10986,
"size" : 2147529933,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 39107,
"entries" : 10401,
"size" : 2147493363,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 40004,
"entries" : 9552,
"size" : 2147523525,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 40107,
"entries" : 8758,
"size" : 2147642891,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 40162,
"entries" : 8166,
"size" : 2147705710,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 40220,
"entries" : 7950,
"size" : 2147521252,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 40278,
"entries" : 6441,
"size" : 1837441502,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 41884,
"entries" : 89,
"size" : 28558495,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 46918,
"entries" : 7614,
"size" : 2001667067,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 49269,
"entries" : 7,
"size" : 1822657,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 52588,
"entries" : 8417,
"size" : 2147659070,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 60509,
"entries" : 8018,
"size" : 2147506406,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 60548,
"entries" : 7581,
"size" : 2147563571,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 60564,
"entries" : 7665,
"size" : 2147490550,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 60596,
"entries" : 7899,
"size" : 2147558053,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 60612,
"entries" : 8252,
"size" : 2147647327,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 60630,
"entries" : 6696,
"size" : 2147853524,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 60654,
"entries" : 6757,
"size" : 2147542237,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 60674,
"entries" : 6604,
"size" : 2147883336,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 60702,
"entries" : 6507,
"size" : 2147548222,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 60727,
"entries" : 6412,
"size" : 2147500437,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 61763,
"entries" : 0,
"size" : 0,
"offloaded" : false,
"underReplicated" : false
} ],
"cursors" : {
"subscription_name" : {
"markDeletePosition" : "61763:932",
"readPosition" : "61763:933",
"waitingReadOp" : true,
"pendingReadOps" : 0,
"messagesConsumedCounter" : 81741,
"cursorLedger" : 60757,
"cursorLedgerLastEntry" : 1246,
"individuallyDeletedMessages" : "[]",
"lastLedgerSwitchTimestamp" : "2022-07-15T15:27:24.981+08:00",
"state" : "Open",
"numberOfEntriesSinceFirstNotAckedMessage" : 1,
"totalNonContiguousDeletedMessagesRange" : 0,
"subscriptionHavePendingRead" : true,
"subscriptionHavePendingReplayRead" : false,
"properties" : { }
},
"perf-test" : {
"markDeletePosition" : "61763:932",
"readPosition" : "61763:933",
"waitingReadOp" : true,
"pendingReadOps" : 0,
"messagesConsumedCounter" : 81741,
"cursorLedger" : 60490,
"cursorLedgerLastEntry" : 14325,
"individuallyDeletedMessages" : "[]",
"lastLedgerSwitchTimestamp" : "2022-07-15T11:49:26.232+08:00",
"state" : "Open",
"numberOfEntriesSinceFirstNotAckedMessage" : 1,
"totalNonContiguousDeletedMessagesRange" : 0,
"subscriptionHavePendingRead" : true,
"subscriptionHavePendingReplayRead" : false,
"properties" : { }
}
},
"schemaLedgers" : [ ],
"compactedLedger" : {
"ledgerId" : -1,
"entries" : -1,
"size" : -1,
"offloaded" : false,
"underReplicated" : false
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on issue #16625: some consumers of several topics can't consume normally
Posted by GitBox <gi...@apache.org>.
lhotari commented on issue #16625:
URL: https://github.com/apache/pulsar/issues/16625#issuecomment-1185486255
@KyrieXu11 Do you have a chance to upgrade to Pulsar 2.10.1 (and the Bookkeeper version that it comes with)?
In 2.10.1 there's a fix in the bookkeeper client (used in the broker), that is https://github.com/apache/bookkeeper/pull/3110 .
If you aren't able to upgrade (or upgrading doesn't fix the issue), Would you be able to test if adding a system property `-Dio.netty.recycler.maxCapacityPerThread=0` helps mitigate the issue (add this to brokers, bookies and the client one at a time to see if it resolves the issue)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on issue #16625: some consumers of several topics can't consume normally
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #16625:
URL: https://github.com/apache/pulsar/issues/16625#issuecomment-1214538737
The issue had no activity for 30 days, mark with Stale label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] smthlow commented on issue #16625: some consumers of several topics can't consume normally
Posted by GitBox <gi...@apache.org>.
smthlow commented on issue #16625:
URL: https://github.com/apache/pulsar/issues/16625#issuecomment-1240620048
Hello, would you mind telling me where should I put `-Dio.netty.recycler.maxCapacityPerThread=0` in order to mitigate this issue? I'm trying to use `apachepulsar/pulsar:2.10.1` Docker image with `bin/pulsar standalone`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on issue #16625: some consumers of several topics can't consume normally
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #16625:
URL: https://github.com/apache/pulsar/issues/16625#issuecomment-1272713273
The issue had no activity for 30 days, mark with Stale label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org