You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/03/26 09:34:31 UTC
[GitHub] [pulsar] codelipenghui opened a new issue #14889: All the consumers get stuck for a broker after enabled transaction
codelipenghui opened a new issue #14889:
URL: https://github.com/apache/pulsar/issues/14889
**Describe the bug**
The pulsar cluster has 3 brokers and 3 bookies, and the broker enables the transaction.
When I use a program to publish and consume messages with transactions on a partitioned topic with 10 partitions
The consumer stats shows only 3 partitions works well for consumption
```
17:18:22.639 [pulsar-timer-5-1:org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl@148] INFO org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl - [persistent://public/default/s_topic-partition-2] [sub] [07360] Prefetched messages: 0 --- Consume throughput received: 4.87 msgs/s --- 0.00 Mbit/s --- Ack sent rate: 4.87 ack/s --- Failed messages: 0 --- batch messages: 0 ---Failed acks: 0
17:18:22.652 [pulsar-timer-5-1:org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl@148] INFO org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl - [persistent://public/default/s_topic-partition-6] [sub] [07360] Prefetched messages: 0 --- Consume throughput received: 5.53 msgs/s --- 0.00 Mbit/s --- Ack sent rate: 5.67 ack/s --- Failed messages: 0 --- batch messages: 0 ---Failed acks: 0
17:18:22.677 [pulsar-timer-5-1:org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl@148] INFO org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl - [persistent://public/default/s_topic-partition-8] [sub] [07360] Prefetched messages: 0 --- Consume throughput received: 4.77 msgs/s --- 0.00 Mbit/s --- Ack sent rate: 4.83 ack/s --- Failed messages: 0 --- batch messages: 0 ---Failed acks: 0
17:18:22.726 [pulsar-timer-5-1:org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl@148] INFO org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl - [persistent://public/default/s_topic-partition-2] [sub] [0767a] Prefetched messages: 0 --- Consume throughput received: 4.60 msgs/s --- 0.00 Mbit/s --- Ack sent rate: 4.67 ack/s --- Failed messages: 0 --- batch messages: 0 ---Failed acks: 0
17:18:22.752 [pulsar-timer-5-1:org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl@148] INFO org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl - [persistent://public/default/s_topic-partition-6] [sub] [0767a] Prefetched messages: 0 --- Consume throughput received: 5.37 msgs/s --- 0.00 Mbit/s --- Ack sent rate: 5.37 ack/s --- Failed messages: 0 --- batch messages: 0 ---Failed acks: 0
17:18:22.772 [pulsar-timer-5-1:org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl@148] INFO org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl - [persistent://public/default/s_topic-partition-8] [sub] [0767a] Prefetched messages: 0 --- Consume throughput received: 4.87 msgs/s --- 0.00 Mbit/s --- Ack sent rate: 5.33 ack/s --- Failed messages: 0 --- batch messages: 0 ---Failed acks: 0
17:18:22.813 [pulsar-timer-5-1:org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl@148] INFO org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl - [persistent://public/default/s_topic-partition-2] [sub] [309f0] Prefetched messages: 0 --- Consume throughput received: 4.60 msgs/s --- 0.00 Mbit/s --- Ack sent rate: 4.60 ack/s --- Failed messages: 0 --- batch messages: 0 ---Failed acks: 0
17:18:22.821 [pulsar-timer-5-1:org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl@148] INFO org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl - [persistent://public/default/s_topic-partition-6] [sub] [309f0] Prefetched messages: 0 --- Consume throughput received: 4.60 msgs/s --- 0.00 Mbit/s --- Ack sent rate: 4.60 ack/s --- Failed messages: 0 --- batch messages: 0 ---Failed acks: 0
17:18:22.874 [pulsar-timer-5-1:org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl@148] INFO org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl - [persistent://public/default/s_topic-partition-8] [sub] [309f0] Prefetched messages: 0 --- Consume throughput received: 5.07 msgs/s --- 0.00 Mbit/s --- Ack sent rate: 5.07 ack/s --- Failed messages: 0 --- batch messages: 0 ---Failed acks: 0
17:18:22.903 [pulsar-timer-5-1:org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl@148] INFO org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl - [persistent://public/default/s_topic-partition-2] [sub] [7d455] Prefetched messages: 0 --- Consume throughput received: 5.03 msgs/s --- 0.00 Mbit/s --- Ack sent rate: 5.03 ack/s --- Failed messages: 0 --- batch messages: 0 ---Failed acks: 0
17:18:22.909 [pulsar-timer-5-1:org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl@148] INFO org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl - [persistent://public/default/s_topic-partition-6] [sub] [7d455] Prefetched messages: 0 --- Consume throughput received: 5.27 msgs/s --- 0.00 Mbit/s --- Ack sent rate: 5.27 ack/s --- Failed messages: 0 --- batch messages: 0 ---Failed acks: 0
17:18:22.912 [pulsar-timer-5-1:org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl@148] INFO org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl - [persistent://public/default/s_topic-partition-8] [sub] [7d455] Prefetched messages: 0 --- Consume throughput received: 4.93 msgs/s --- 0.00 Mbit/s --- Ack sent rate: 4.93 ack/s --- Failed messages: 0 --- batch messages: 0 ---Failed acks: 0
17:18:22.945 [pulsar-timer-5-1:org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl@148] INFO org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl - [persistent://public/default/s_topic-partition-2] [sub] [d54cd] Prefetched messages: 0 --- Consume throughput received: 4.87 msgs/s --- 0.00 Mbit/s --- Ack sent rate: 4.87 ack/s --- Failed messages: 0 --- batch messages: 0 ---Failed acks: 0
17:18:22.957 [pulsar-timer-5-1:org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl@148] INFO org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl - [persistent://public/default/s_topic-partition-6] [sub] [d54cd] Prefetched messages: 0 --- Consume throughput received: 5.13 msgs/s --- 0.00 Mbit/s --- Ack sent rate: 5.13 ack/s --- Failed messages: 0 --- batch messages: 0 ---Failed acks: 0
17:18:22.969 [pulsar-timer-5-1:org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl@148] INFO org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl - [persistent://public/default/s_topic-partition-8] [sub] [d54cd] Prefetched messages: 0 --- Consume throughput received: 5.07 msgs/s --- 0.00 Mbit/s --- Ack sent rate: 5.07 ack/s --- Failed messages: 0 --- batch messages: 0 ---Failed acks: 0
```
After checking the topic ownership
```
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
~/Downloads/geo-cluster/cluster-a (branch-0.8.1*) » bin/pulsar-admin topics lookup s_topic-partition-0 lipenghui@lipenghuideMacBook-Pro-2
Warning: Nashorn engine is planned to be removed from a future JDK release
"pulsar://127.0.0.1:6650"
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
~/Downloads/geo-cluster/cluster-a (branch-0.8.1*) » bin/pulsar-admin topics lookup s_topic-partition-1 lipenghui@lipenghuideMacBook-Pro-2
Warning: Nashorn engine is planned to be removed from a future JDK release
"pulsar://127.0.0.1:6651"
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
~/Downloads/geo-cluster/cluster-a (branch-0.8.1*) » bin/pulsar-admin topics lookup s_topic-partition-2 lipenghui@lipenghuideMacBook-Pro-2
Warning: Nashorn engine is planned to be removed from a future JDK release
"pulsar://127.0.0.1:6652"
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
~/Downloads/geo-cluster/cluster-a (branch-0.8.1*) » bin/pulsar-admin topics lookup s_topic-partition-3 lipenghui@lipenghuideMacBook-Pro-2
Warning: Nashorn engine is planned to be removed from a future JDK release
"pulsar://127.0.0.1:6650"
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
~/Downloads/geo-cluster/cluster-a (branch-0.8.1*) » bin/pulsar-admin topics lookup s_topic-partition-4 lipenghui@lipenghuideMacBook-Pro-2
Warning: Nashorn engine is planned to be removed from a future JDK release
"pulsar://127.0.0.1:6650"
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
~/Downloads/geo-cluster/cluster-a (branch-0.8.1*) » bin/pulsar-admin topics lookup s_topic-partition-5 lipenghui@lipenghuideMacBook-Pro-2
Warning: Nashorn engine is planned to be removed from a future JDK release
"pulsar://127.0.0.1:6651"
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
~/Downloads/geo-cluster/cluster-a (branch-0.8.1*) » bin/pulsar-admin topics lookup s_topic-partition-6 lipenghui@lipenghuideMacBook-Pro-2
Warning: Nashorn engine is planned to be removed from a future JDK release
"pulsar://127.0.0.1:6652"
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
~/Downloads/geo-cluster/cluster-a (branch-0.8.1*) » bin/pulsar-admin topics lookup s_topic-partition-7 lipenghui@lipenghuideMacBook-Pro-2
Warning: Nashorn engine is planned to be removed from a future JDK release
"pulsar://127.0.0.1:6650"
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
~/Downloads/geo-cluster/cluster-a (branch-0.8.1*) » bin/pulsar-admin topics lookup s_topic-partition-8 lipenghui@lipenghuideMacBook-Pro-2
Warning: Nashorn engine is planned to be removed from a future JDK release
"pulsar://127.0.0.1:6650"
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
~/Downloads/geo-cluster/cluster-a (branch-0.8.1*) » bin/pulsar-admin topics lookup s_topic-partition-9 lipenghui@lipenghuideMacBook-Pro-2
Warning: Nashorn engine is planned to be removed from a future JDK release
"pulsar://127.0.0.1:6651"
```
You can see partition 2,6,8 is owned by `pulsar://127.0.0.1:6652`. Other partitions are distributed to `pulsar://127.0.0.1:6650` and `pulsar://127.0.0.1:6651`, but all of them get stucked.
The internal stats and stats:
```
~/Downloads/geo-cluster/cluster-a-1 (branch-0.8.1*) » bin/pulsar-admin topics stats s_topic-partition-0 lipenghui@lipenghuideMacBook-Pro-2
Warning: Nashorn engine is planned to be removed from a future JDK release
{
"msgRateIn" : 49.6341807476304,
"msgThroughputIn" : 2779.5141218673025,
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"bytesInCounter" : 22151660,
"msgInCounter" : 401744,
"bytesOutCounter" : 1333622,
"msgOutCounter" : 24921,
"averageMsgSize" : 56.0,
"msgChunkPublished" : false,
"storageSize" : 20814910,
"backlogSize" : 20814910,
"offloadedStorageSize" : 0,
"lastOffloadLedgerId" : 0,
"lastOffloadSuccessTimeStamp" : 0,
"lastOffloadFailureTimeStamp" : 0,
"publishers" : [ {
"accessMode" : "Shared",
"msgRateIn" : 4.966751606728177,
"msgThroughputIn" : 278.13808997677796,
"averageMsgSize" : 56.0,
"chunkedMessageRate" : 0.0,
"producerId" : 10,
"metadata" : { },
"connectedSince" : "2022-03-26T14:50:22.57512+08:00",
"clientVersion" : "2.10.0-SNAPSHOT",
"producerName" : "cluster-a-0-37",
"address" : "/127.0.0.1:50789"
}, {
"accessMode" : "Shared",
"msgRateIn" : 4.966751588764786,
"msgThroughputIn" : 278.138088970828,
"averageMsgSize" : 56.0,
"chunkedMessageRate" : 0.0,
"producerId" : 11,
"metadata" : { },
"connectedSince" : "2022-03-26T14:50:22.579051+08:00",
"clientVersion" : "2.10.0-SNAPSHOT",
"producerName" : "cluster-a-0-38",
"address" : "/127.0.0.1:50789"
}, {
"accessMode" : "Shared",
"msgRateIn" : 4.966751579658919,
"msgThroughputIn" : 278.13808846089944,
"averageMsgSize" : 56.0,
"chunkedMessageRate" : 0.0,
"producerId" : 12,
"metadata" : { },
"connectedSince" : "2022-03-26T14:50:22.577037+08:00",
"clientVersion" : "2.10.0-SNAPSHOT",
"producerName" : "cluster-a-0-39",
"address" : "/127.0.0.1:50789"
}, {
"accessMode" : "Shared",
"msgRateIn" : 4.966751137776072,
"msgThroughputIn" : 278.13806371546,
"averageMsgSize" : 56.0,
"chunkedMessageRate" : 0.0,
"producerId" : 17,
"metadata" : { },
"connectedSince" : "2022-03-26T14:50:22.585829+08:00",
"clientVersion" : "2.10.0-SNAPSHOT",
"producerName" : "cluster-a-0-44",
"address" : "/127.0.0.1:50789"
}, {
"accessMode" : "Shared",
"msgRateIn" : 4.933417219186362,
"msgThroughputIn" : 276.2713642744362,
"averageMsgSize" : 56.0,
"chunkedMessageRate" : 0.0,
"producerId" : 18,
"metadata" : { },
"connectedSince" : "2022-03-26T14:50:22.593333+08:00",
"clientVersion" : "2.10.0-SNAPSHOT",
"producerName" : "cluster-a-0-45",
"address" : "/127.0.0.1:50789"
}, {
"accessMode" : "Shared",
"msgRateIn" : 4.966751083472002,
"msgThroughputIn" : 278.13806067443215,
"averageMsgSize" : 56.0,
"chunkedMessageRate" : 0.0,
"producerId" : 19,
"metadata" : { },
"connectedSince" : "2022-03-26T14:50:22.589781+08:00",
"clientVersion" : "2.10.0-SNAPSHOT",
"producerName" : "cluster-a-0-46",
"address" : "/127.0.0.1:50789"
}, {
"accessMode" : "Shared",
"msgRateIn" : 4.9667516437311106,
"msgThroughputIn" : 278.1380920489422,
"averageMsgSize" : 56.0,
"chunkedMessageRate" : 0.0,
"producerId" : 13,
"metadata" : { },
"connectedSince" : "2022-03-26T14:50:22.578721+08:00",
"clientVersion" : "2.10.0-SNAPSHOT",
"producerName" : "cluster-a-0-40",
"address" : "/127.0.0.1:50789"
}, {
"accessMode" : "Shared",
"msgRateIn" : 4.9667516367775395,
"msgThroughputIn" : 278.1380916595422,
"averageMsgSize" : 56.0,
"chunkedMessageRate" : 0.0,
"producerId" : 14,
"metadata" : { },
"connectedSince" : "2022-03-26T14:50:22.580607+08:00",
"clientVersion" : "2.10.0-SNAPSHOT",
"producerName" : "cluster-a-0-41",
"address" : "/127.0.0.1:50789"
}, {
"accessMode" : "Shared",
"msgRateIn" : 4.966751629327285,
"msgThroughputIn" : 278.1380912423279,
"averageMsgSize" : 56.0,
"chunkedMessageRate" : 0.0,
"producerId" : 15,
"metadata" : { },
"connectedSince" : "2022-03-26T14:50:22.582231+08:00",
"clientVersion" : "2.10.0-SNAPSHOT",
"producerName" : "cluster-a-0-42",
"address" : "/127.0.0.1:50789"
}, {
"accessMode" : "Shared",
"msgRateIn" : 4.966751622208152,
"msgThroughputIn" : 278.1380908436565,
"averageMsgSize" : 56.0,
"chunkedMessageRate" : 0.0,
"producerId" : 16,
"metadata" : { },
"connectedSince" : "2022-03-26T14:50:22.583925+08:00",
"clientVersion" : "2.10.0-SNAPSHOT",
"producerName" : "cluster-a-0-43",
"address" : "/127.0.0.1:50789"
} ],
"waitingPublishers" : 0,
"subscriptions" : {
"sub" : {
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"bytesOutCounter" : 1333622,
"msgOutCounter" : 24921,
"msgRateRedeliver" : 0.0,
"chunkedMessageRate" : 0,
"msgBacklog" : 376760,
"backlogSize" : 0,
"msgBacklogNoDelayed" : 376760,
"blockedSubscriptionOnUnackedMsgs" : false,
"msgDelayed" : 0,
"unackedMessages" : 0,
"type" : "Shared",
"msgRateExpired" : 0.0,
"totalMsgExpired" : 0,
"lastExpireTimestamp" : 0,
"lastConsumedFlowTimestamp" : 1648277422510,
"lastConsumedTimestamp" : 0,
"lastAckedTimestamp" : 0,
"lastMarkDeleteAdvancedTimestamp" : 1648276627087,
"consumers" : [ {
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"bytesOutCounter" : 0,
"msgOutCounter" : 0,
"msgRateRedeliver" : 0.0,
"chunkedMessageRate" : 0.0,
"consumerName" : "2a2ae",
"availablePermits" : 1000,
"unackedMessages" : 0,
"avgMessagesPerEntry" : 0,
"blockedConsumerOnUnackedMsgs" : false,
"lastAckedTimestamp" : 0,
"lastConsumedTimestamp" : 0,
"metadata" : { },
"connectedSince" : "2022-03-26T14:50:21.096862+08:00",
"clientVersion" : "2.10.0-SNAPSHOT",
"address" : "/127.0.0.1:50789"
}, {
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"bytesOutCounter" : 0,
"msgOutCounter" : 0,
"msgRateRedeliver" : 0.0,
"chunkedMessageRate" : 0.0,
"consumerName" : "6b4c9",
"availablePermits" : 1000,
"unackedMessages" : 0,
"avgMessagesPerEntry" : 0,
"blockedConsumerOnUnackedMsgs" : false,
"lastAckedTimestamp" : 0,
"lastConsumedTimestamp" : 0,
"metadata" : { },
"connectedSince" : "2022-03-26T14:50:21.216076+08:00",
"clientVersion" : "2.10.0-SNAPSHOT",
"address" : "/127.0.0.1:50789"
}, {
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"bytesOutCounter" : 0,
"msgOutCounter" : 0,
"msgRateRedeliver" : 0.0,
"chunkedMessageRate" : 0.0,
"consumerName" : "6c798",
"availablePermits" : 1000,
"unackedMessages" : 0,
"avgMessagesPerEntry" : 0,
"blockedConsumerOnUnackedMsgs" : false,
"lastAckedTimestamp" : 0,
"lastConsumedTimestamp" : 0,
"metadata" : { },
"connectedSince" : "2022-03-26T14:50:21.864061+08:00",
"clientVersion" : "2.10.0-SNAPSHOT",
"address" : "/127.0.0.1:50789"
}, {
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"bytesOutCounter" : 0,
"msgOutCounter" : 0,
"msgRateRedeliver" : 0.0,
"chunkedMessageRate" : 0.0,
"consumerName" : "9dda8",
"availablePermits" : 1000,
"unackedMessages" : 0,
"avgMessagesPerEntry" : 0,
"blockedConsumerOnUnackedMsgs" : false,
"lastAckedTimestamp" : 0,
"lastConsumedTimestamp" : 0,
"metadata" : { },
"connectedSince" : "2022-03-26T14:50:21.968646+08:00",
"clientVersion" : "2.10.0-SNAPSHOT",
"address" : "/127.0.0.1:50789"
}, {
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"bytesOutCounter" : 0,
"msgOutCounter" : 0,
"msgRateRedeliver" : 0.0,
"chunkedMessageRate" : 0.0,
"consumerName" : "7bf7b",
"availablePermits" : 1000,
"unackedMessages" : 0,
"avgMessagesPerEntry" : 0,
"blockedConsumerOnUnackedMsgs" : false,
"lastAckedTimestamp" : 0,
"lastConsumedTimestamp" : 0,
"metadata" : { },
"connectedSince" : "2022-03-26T14:50:22.113426+08:00",
"clientVersion" : "2.10.0-SNAPSHOT",
"address" : "/127.0.0.1:50789"
}, {
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"bytesOutCounter" : 0,
"msgOutCounter" : 0,
"msgRateRedeliver" : 0.0,
"chunkedMessageRate" : 0.0,
"consumerName" : "07360",
"availablePermits" : 1000,
"unackedMessages" : 0,
"avgMessagesPerEntry" : 0,
"blockedConsumerOnUnackedMsgs" : false,
"lastAckedTimestamp" : 0,
"lastConsumedTimestamp" : 0,
"metadata" : { },
"connectedSince" : "2022-03-26T14:50:22.218797+08:00",
"clientVersion" : "2.10.0-SNAPSHOT",
"address" : "/127.0.0.1:50789"
}, {
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"bytesOutCounter" : 0,
"msgOutCounter" : 0,
"msgRateRedeliver" : 0.0,
"chunkedMessageRate" : 0.0,
"consumerName" : "0767a",
"availablePermits" : 1000,
"unackedMessages" : 0,
"avgMessagesPerEntry" : 0,
"blockedConsumerOnUnackedMsgs" : false,
"lastAckedTimestamp" : 0,
"lastConsumedTimestamp" : 0,
"metadata" : { },
"connectedSince" : "2022-03-26T14:50:22.309638+08:00",
"clientVersion" : "2.10.0-SNAPSHOT",
"address" : "/127.0.0.1:50789"
}, {
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"bytesOutCounter" : 0,
"msgOutCounter" : 0,
"msgRateRedeliver" : 0.0,
"chunkedMessageRate" : 0.0,
"consumerName" : "309f0",
"availablePermits" : 1000,
"unackedMessages" : 0,
"avgMessagesPerEntry" : 0,
"blockedConsumerOnUnackedMsgs" : false,
"lastAckedTimestamp" : 0,
"lastConsumedTimestamp" : 0,
"metadata" : { },
"connectedSince" : "2022-03-26T14:50:22.377575+08:00",
"clientVersion" : "2.10.0-SNAPSHOT",
"address" : "/127.0.0.1:50789"
}, {
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"bytesOutCounter" : 0,
"msgOutCounter" : 0,
"msgRateRedeliver" : 0.0,
"chunkedMessageRate" : 0.0,
"consumerName" : "7d455",
"availablePermits" : 1000,
"unackedMessages" : 0,
"avgMessagesPerEntry" : 0,
"blockedConsumerOnUnackedMsgs" : false,
"lastAckedTimestamp" : 0,
"lastConsumedTimestamp" : 0,
"metadata" : { },
"connectedSince" : "2022-03-26T14:50:22.44303+08:00",
"clientVersion" : "2.10.0-SNAPSHOT",
"address" : "/127.0.0.1:50789"
}, {
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"bytesOutCounter" : 0,
"msgOutCounter" : 0,
"msgRateRedeliver" : 0.0,
"chunkedMessageRate" : 0.0,
"consumerName" : "d54cd",
"availablePermits" : 1000,
"unackedMessages" : 0,
"avgMessagesPerEntry" : 0,
"blockedConsumerOnUnackedMsgs" : false,
"lastAckedTimestamp" : 0,
"lastConsumedTimestamp" : 0,
"metadata" : { },
"connectedSince" : "2022-03-26T14:50:22.496425+08:00",
"clientVersion" : "2.10.0-SNAPSHOT",
"address" : "/127.0.0.1:50789"
} ],
"isDurable" : true,
"isReplicated" : false,
"allowOutOfOrderDelivery" : false,
"consumersAfterMarkDeletePosition" : { },
"nonContiguousDeletedMessagesRanges" : 0,
"nonContiguousDeletedMessagesRangesSerializedSize" : 18,
"replicated" : false,
"durable" : true
}
},
"replication" : { },
"deduplicationStatus" : "Disabled",
"nonContiguousDeletedMessagesRanges" : 0,
"nonContiguousDeletedMessagesRangesSerializedSize" : 18,
"compaction" : {
"lastCompactionRemovedEventCount" : 0,
"lastCompactionSucceedTimestamp" : 0,
"lastCompactionFailedTimestamp" : 0,
"lastCompactionDurationTimeInMills" : 0
}
}
~/Downloads/geo-cluster/cluster-a-1 (branch-0.8.1*) » bin/pulsar-admin transactions transaction-buffer-stats -t s_topic-partition-0 lipenghui@lipenghuideMacBook-Pro-2
Warning: Nashorn engine is planned to be removed from a future JDK release
{
"state" : "Ready",
"maxReadPosition" : "1203:25701",
"lastSnapshotTimestamps" : 1648285007055
}
~/Downloads/geo-cluster/cluster-a-1 (branch-0.8.1*) » bin/pulsar-admin topics stats-internal s_topic-partition-0 lipenghui@lipenghuideMacBook-Pro-2
Warning: Nashorn engine is planned to be removed from a future JDK release
{
"entriesAddedCounter" : 397642,
"numberOfEntries" : 372860,
"totalSize" : 20596510,
"currentLedgerEntries" : 22860,
"currentLedgerSize" : 1280160,
"lastLedgerCreatedTimestamp" : "2022-03-26T16:48:08.177+08:00",
"lastLedgerCreationFailureTimestamp" : "2022-03-26T14:37:11.86+08:00",
"waitingCursorsCount" : 0,
"pendingAddEntriesCount" : 0,
"lastConfirmedEntry" : "1203:22859",
"state" : "LedgerOpened",
"ledgers" : [ {
"ledgerId" : 860,
"entries" : 50000,
"size" : 2687610,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 921,
"entries" : 50000,
"size" : 2700000,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 974,
"entries" : 50000,
"size" : 2746290,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 1017,
"entries" : 50000,
"size" : 2782450,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 1070,
"entries" : 50000,
"size" : 2800000,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 1097,
"entries" : 50000,
"size" : 2800000,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 1150,
"entries" : 50000,
"size" : 2800000,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 1203,
"entries" : 0,
"size" : 0,
"offloaded" : false,
"underReplicated" : false
} ],
"cursors" : {
"sub" : {
"markDeletePosition" : "669:24781",
"readPosition" : "669:24782",
"waitingReadOp" : false,
"pendingReadOps" : 0,
"messagesConsumedCounter" : 24782,
"cursorLedger" : 681,
"cursorLedgerLastEntry" : 479,
"individuallyDeletedMessages" : "[]",
"lastLedgerSwitchTimestamp" : "2022-03-26T14:28:42.614+08:00",
"state" : "Open",
"numberOfEntriesSinceFirstNotAckedMessage" : 1,
"totalNonContiguousDeletedMessagesRange" : 0,
"subscriptionHavePendingRead" : true,
"subscriptionHavePendingReplayRead" : false,
"properties" : { }
},
"transaction-buffer-sub" : {
"markDeletePosition" : "669:-1",
"readPosition" : "669:0",
"waitingReadOp" : false,
"pendingReadOps" : 0,
"messagesConsumedCounter" : 0,
"cursorLedger" : -1,
"cursorLedgerLastEntry" : -1,
"individuallyDeletedMessages" : "[]",
"lastLedgerSwitchTimestamp" : "2022-03-26T14:28:42.765+08:00",
"state" : "Uninitialized",
"numberOfEntriesSinceFirstNotAckedMessage" : 1,
"totalNonContiguousDeletedMessagesRange" : 0,
"subscriptionHavePendingRead" : false,
"subscriptionHavePendingReplayRead" : false,
"properties" : { }
}
},
"schemaLedgers" : [ {
"ledgerId" : 69,
"entries" : 1,
"size" : 103,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 88,
"entries" : 1,
"size" : 129,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 78,
"entries" : 1,
"size" : 85,
"offloaded" : false,
"underReplicated" : false
} ],
"compactedLedger" : {
"ledgerId" : -1,
"entries" : -1,
"size" : -1,
"offloaded" : false,
"underReplicated" : false
}
}
```
From the topic stats, looks like the consumer connects to the topic and never gets any messages, the `msgOutCounter` is 0, the `lastConsumedFlowTimestamp` is `2022-03-26 14:50:22` and the connect time is `2022-03-26T14:50:21.096862+08:00`.
From the broker side:
```
14:50:21.092 [pulsar-io-16-9] INFO org.apache.pulsar.broker.service.ServerCnx - [/127.0.0.1:50789] Subscribing on topic persistent://public/default/s_topic-partition-0 / sub
14:50:21.097 [ForkJoinPool.commonPool-worker-25] INFO org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://public/default/s_topic-partition-0] There are no replicated subscriptions on the topic
14:50:21.097 [ForkJoinPool.commonPool-worker-25] INFO org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://public/default/s_topic-partition-0][sub] Created new subscription for 0
14:50:21.097 [ForkJoinPool.commonPool-worker-25] INFO org.apache.pulsar.broker.service.ServerCnx - [/127.0.0.1:50789] Created subscription on topic persistent://public/default/s_topic-partition-0 / sub
14:50:21.207 [pulsar-io-16-9] INFO org.apache.pulsar.broker.service.ServerCnx - [/127.0.0.1:50789] Subscribing on topic persistent://public/default/s_topic-partition-0 / sub
14:50:21.216 [ForkJoinPool.commonPool-worker-25] INFO org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://public/default/s_topic-partition-0] There are no replicated subscriptions on the topic
14:50:21.216 [ForkJoinPool.commonPool-worker-25] INFO org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://public/default/s_topic-partition-0][sub] Created new subscription for 10
14:50:21.216 [ForkJoinPool.commonPool-worker-25] INFO org.apache.pulsar.broker.service.ServerCnx - [/127.0.0.1:50789] Created subscription on topic persistent://public/default/s_topic-partition-0 / sub
14:50:21.859 [pulsar-io-16-9] INFO org.apache.pulsar.broker.service.ServerCnx - [/127.0.0.1:50789] Subscribing on topic persistent://public/default/s_topic-partition-0 / sub
14:50:21.864 [ForkJoinPool.commonPool-worker-25] INFO org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://public/default/s_topic-partition-0] There are no replicated subscriptions on the topic
14:50:21.864 [ForkJoinPool.commonPool-worker-25] INFO org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://public/default/s_topic-partition-0][sub] Created new subscription for 20
14:50:21.864 [ForkJoinPool.commonPool-worker-25] INFO org.apache.pulsar.broker.service.ServerCnx - [/127.0.0.1:50789] Created subscription on topic persistent://public/default/s_topic-partition-0 / sub
14:50:21.965 [pulsar-io-16-9] INFO org.apache.pulsar.broker.service.ServerCnx - [/127.0.0.1:50789] Subscribing on topic persistent://public/default/s_topic-partition-0 / sub
14:50:21.968 [ForkJoinPool.commonPool-worker-25] INFO org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://public/default/s_topic-partition-0] There are no replicated subscriptions on the topic
14:50:21.968 [ForkJoinPool.commonPool-worker-25] INFO org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://public/default/s_topic-partition-0][sub] Created new subscription for 30
14:50:21.968 [ForkJoinPool.commonPool-worker-25] INFO org.apache.pulsar.broker.service.ServerCnx - [/127.0.0.1:50789] Created subscription on topic persistent://public/default/s_topic-partition-0 / sub
14:50:22.109 [pulsar-io-16-9] INFO org.apache.pulsar.broker.service.ServerCnx - [/127.0.0.1:50789] Subscribing on topic persistent://public/default/s_topic-partition-0 / sub
14:50:22.113 [ForkJoinPool.commonPool-worker-25] INFO org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://public/default/s_topic-partition-0] There are no replicated subscriptions on the topic
14:50:22.113 [ForkJoinPool.commonPool-worker-25] INFO org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://public/default/s_topic-partition-0][sub] Created new subscription for 40
14:50:22.113 [ForkJoinPool.commonPool-worker-25] INFO org.apache.pulsar.broker.service.ServerCnx - [/127.0.0.1:50789] Created subscription on topic persistent://public/default/s_topic-partition-0 / sub
14:50:22.203 [pulsar-io-16-9] INFO org.apache.pulsar.broker.service.ServerCnx - [/127.0.0.1:50789] Subscribing on topic persistent://public/default/s_topic-partition-0 / sub
14:50:22.218 [ForkJoinPool.commonPool-worker-25] INFO org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://public/default/s_topic-partition-0] There are no replicated subscriptions on the topic
14:50:22.218 [ForkJoinPool.commonPool-worker-25] INFO org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://public/default/s_topic-partition-0][sub] Created new subscription for 50
14:50:22.218 [ForkJoinPool.commonPool-worker-25] INFO org.apache.pulsar.broker.service.ServerCnx - [/127.0.0.1:50789] Created subscription on topic persistent://public/default/s_topic-partition-0 / sub
14:50:22.291 [pulsar-io-16-9] INFO org.apache.pulsar.broker.service.ServerCnx - [/127.0.0.1:50789] Subscribing on topic persistent://public/default/s_topic-partition-0 / sub
14:50:22.309 [ForkJoinPool.commonPool-worker-25] INFO org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://public/default/s_topic-partition-0] There are no replicated subscriptions on the topic
14:50:22.309 [ForkJoinPool.commonPool-worker-25] INFO org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://public/default/s_topic-partition-0][sub] Created new subscription for 60
14:50:22.309 [ForkJoinPool.commonPool-worker-25] INFO org.apache.pulsar.broker.service.ServerCnx - [/127.0.0.1:50789] Created subscription on topic persistent://public/default/s_topic-partition-0 / sub
14:50:22.373 [pulsar-io-16-9] INFO org.apache.pulsar.broker.service.ServerCnx - [/127.0.0.1:50789] Subscribing on topic persistent://public/default/s_topic-partition-0 / sub
14:50:22.377 [ForkJoinPool.commonPool-worker-25] INFO org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://public/default/s_topic-partition-0] There are no replicated subscriptions on the topic
14:50:22.377 [ForkJoinPool.commonPool-worker-25] INFO org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://public/default/s_topic-partition-0][sub] Created new subscription for 70
14:50:22.377 [ForkJoinPool.commonPool-worker-25] INFO org.apache.pulsar.broker.service.ServerCnx - [/127.0.0.1:50789] Created subscription on topic persistent://public/default/s_topic-partition-0 / sub
14:50:22.433 [pulsar-io-16-9] INFO org.apache.pulsar.broker.service.ServerCnx - [/127.0.0.1:50789] Subscribing on topic persistent://public/default/s_topic-partition-0 / sub
14:50:22.443 [ForkJoinPool.commonPool-worker-25] INFO org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://public/default/s_topic-partition-0] There are no replicated subscriptions on the topic
14:50:22.443 [ForkJoinPool.commonPool-worker-25] INFO org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://public/default/s_topic-partition-0][sub] Created new subscription for 80
14:50:22.443 [ForkJoinPool.commonPool-worker-25] INFO org.apache.pulsar.broker.service.ServerCnx - [/127.0.0.1:50789] Created subscription on topic persistent://public/default/s_topic-partition-0 / sub
14:50:22.490 [pulsar-io-16-9] INFO org.apache.pulsar.broker.service.ServerCnx - [/127.0.0.1:50789] Subscribing on topic persistent://public/default/s_topic-partition-0 / sub
14:50:22.496 [ForkJoinPool.commonPool-worker-25] INFO org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://public/default/s_topic-partition-0] There are no replicated subscriptions on the topic
14:50:22.496 [ForkJoinPool.commonPool-worker-25] INFO org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://public/default/s_topic-partition-0][sub] Created new subscription for 90
14:50:22.496 [ForkJoinPool.commonPool-worker-25] INFO org.apache.pulsar.broker.service.ServerCnx - [/127.0.0.1:50789] Created subscription on topic persistent://public/default/s_topic-partition-0 / sub
14:50:22.564 [pulsar-io-16-9] INFO org.apache.pulsar.broker.service.ServerCnx - [/127.0.0.1:50789][persistent://public/default/s_topic-partition-0] Creating producer. producerId=10
14:50:22.565 [pulsar-io-16-9] INFO org.apache.pulsar.broker.service.ServerCnx - [/127.0.0.1:50789][persistent://public/default/s_topic-partition-0] Creating producer. producerId=11
14:50:22.567 [pulsar-io-16-9] INFO org.apache.pulsar.broker.service.ServerCnx - [/127.0.0.1:50789][persistent://public/default/s_topic-partition-0] Creating producer. producerId=12
14:50:22.569 [pulsar-io-16-9] INFO org.apache.pulsar.broker.service.ServerCnx - [/127.0.0.1:50789][persistent://public/default/s_topic-partition-0] Creating producer. producerId=13
14:50:22.571 [pulsar-io-16-9] INFO org.apache.pulsar.broker.service.ServerCnx - [/127.0.0.1:50789][persistent://public/default/s_topic-partition-0] Creating producer. producerId=14
14:50:22.572 [pulsar-io-16-9] INFO org.apache.pulsar.broker.service.ServerCnx - [/127.0.0.1:50789][persistent://public/default/s_topic-partition-0] Creating producer. producerId=15
14:50:22.572 [pulsar-io-16-9] INFO org.apache.pulsar.broker.service.ServerCnx - [/127.0.0.1:50789][persistent://public/default/s_topic-partition-0] Creating producer. producerId=16
14:50:22.574 [pulsar-io-16-9] INFO org.apache.pulsar.broker.service.ServerCnx - [/127.0.0.1:50789][persistent://public/default/s_topic-partition-0] Creating producer. producerId=17
14:50:22.574 [pulsar-io-16-9] INFO org.apache.pulsar.broker.service.ServerCnx - [/127.0.0.1:50789][persistent://public/default/s_topic-partition-0] Creating producer. producerId=18
14:50:22.574 [pulsar-io-16-9] INFO org.apache.pulsar.broker.service.ServerCnx - [/127.0.0.1:50789][persistent://public/default/s_topic-partition-0] Creating producer. producerId=19
14:50:22.575 [BookKeeperClientWorker-OrderedExecutor-9-0] INFO org.apache.pulsar.broker.service.ServerCnx - [/127.0.0.1:50789] Created new producer: Producer{topic=PersistentTopic{topic=persistent://public/default/s_topic-partition-0}, client=/127.0.0.1:50789, producerName=cluster-a-0-37, producerId=10}
14:50:22.577 [BookKeeperClientWorker-OrderedExecutor-4-0] INFO org.apache.pulsar.broker.service.ServerCnx - [/127.0.0.1:50789] Created new producer: Producer{topic=PersistentTopic{topic=persistent://public/default/s_topic-partition-0}, client=/127.0.0.1:50789, producerName=cluster-a-0-39, producerId=12}
14:50:22.578 [BookKeeperClientWorker-OrderedExecutor-4-0] INFO org.apache.pulsar.broker.service.ServerCnx - [/127.0.0.1:50789] Created new producer: Producer{topic=PersistentTopic{topic=persistent://public/default/s_topic-partition-0}, client=/127.0.0.1:50789, producerName=cluster-a-0-40, producerId=13}
14:50:22.579 [BookKeeperClientWorker-OrderedExecutor-6-0] INFO org.apache.pulsar.broker.service.ServerCnx - [/127.0.0.1:50789] Created new producer: Producer{topic=PersistentTopic{topic=persistent://public/default/s_topic-partition-0}, client=/127.0.0.1:50789, producerName=cluster-a-0-38, producerId=11}
14:50:22.580 [BookKeeperClientWorker-OrderedExecutor-4-0] INFO org.apache.pulsar.broker.service.ServerCnx - [/127.0.0.1:50789] Created new producer: Producer{topic=PersistentTopic{topic=persistent://public/default/s_topic-partition-0}, client=/127.0.0.1:50789, producerName=cluster-a-0-41, producerId=14}
14:50:22.582 [BookKeeperClientWorker-OrderedExecutor-4-0] INFO org.apache.pulsar.broker.service.ServerCnx - [/127.0.0.1:50789] Created new producer: Producer{topic=PersistentTopic{topic=persistent://public/default/s_topic-partition-0}, client=/127.0.0.1:50789, producerName=cluster-a-0-42, producerId=15}
14:50:22.584 [BookKeeperClientWorker-OrderedExecutor-4-0] INFO org.apache.pulsar.broker.service.ServerCnx - [/127.0.0.1:50789] Created new producer: Producer{topic=PersistentTopic{topic=persistent://public/default/s_topic-partition-0}, client=/127.0.0.1:50789, producerName=cluster-a-0-43, producerId=16}
14:50:22.585 [BookKeeperClientWorker-OrderedExecutor-4-0] INFO org.apache.pulsar.broker.service.ServerCnx - [/127.0.0.1:50789] Created new producer: Producer{topic=PersistentTopic{topic=persistent://public/default/s_topic-partition-0}, client=/127.0.0.1:50789, producerName=cluster-a-0-44, producerId=17}
14:50:22.590 [BookKeeperClientWorker-OrderedExecutor-4-0] INFO org.apache.pulsar.broker.service.ServerCnx - [/127.0.0.1:50789] Created new producer: Producer{topic=PersistentTopic{topic=persistent://public/default/s_topic-partition-0}, client=/127.0.0.1:50789, producerName=cluster-a-0-46, producerId=19}
14:50:22.593 [BookKeeperClientWorker-OrderedExecutor-4-0] INFO org.apache.pulsar.broker.service.ServerCnx - [/127.0.0.1:50789] Created new producer: Producer{topic=PersistentTopic{topic=persistent://public/default/s_topic-partition-0}, client=/127.0.0.1:50789, producerName=cluster-a-0-45, producerId=18}
15:06:04.289 [BookKeeperClientWorker-OrderedExecutor-11-0] INFO org.apache.bookkeeper.mledger.impl.OpAddEntry - [public/default/persistent/s_topic-partition-0] Closing ledger 860 for being full
15:06:04.382 [main-EventThread] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/default/persistent/s_topic-partition-0] Creating a new ledger after closed
15:06:04.401 [BookKeeperClientWorker-OrderedExecutor-4-0] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/default/persistent/s_topic-partition-0] Creating ledger, metadata: {component=[109, 97, 110, 97, 103, 101, 100, 45, 108, 101, 100, 103, 101, 114], pulsar/managed-ledger=[112, 117, 98, 108, 105, 99, 47, 100, 101, 102, 97, 117, 108, 116, 47, 112, 101, 114, 115, 105, 115, 116, 101, 110, 116, 47, 115, 95, 116, 111, 112, 105, 99, 45, 112, 97, 114, 116, 105, 116, 105, 111, 110, 45, 48], application=[112, 117, 108, 115, 97, 114]} - metadata ops timeout : 60 seconds
15:06:04.590 [main-EventThread] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/default/persistent/s_topic-partition-0] Created new ledger 921
15:06:04.647 [bookkeeper-ml-scheduler-OrderedScheduler-7-0] WARN org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - Cursor: ManagedCursorImpl{ledger=public/default/persistent/s_topic-partition-0, name=sub, ackPos=669:24781, readPos=669:24782} does not exist in the managed-ledger.
15:06:04.647 [bookkeeper-ml-scheduler-OrderedScheduler-7-0] WARN org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - Cursor: NonDurableCursorImpl{ledger=public/default/persistent/s_topic-partition-0, ackPos=669:-1, readPos=669:0} does not exist in the managed-ledger.
15:23:05.346 [BookKeeperClientWorker-OrderedExecutor-11-0] INFO org.apache.bookkeeper.mledger.impl.OpAddEntry - [public/default/persistent/s_topic-partition-0] Closing ledger 921 for being full
15:23:05.460 [main-EventThread] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/default/persistent/s_topic-partition-0] Creating a new ledger after closed
15:23:05.490 [BookKeeperClientWorker-OrderedExecutor-1-0] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/default/persistent/s_topic-partition-0] Creating ledger, metadata: {component=[109, 97, 110, 97, 103, 101, 100, 45, 108, 101, 100, 103, 101, 114], pulsar/managed-ledger=[112, 117, 98, 108, 105, 99, 47, 100, 101, 102, 97, 117, 108, 116, 47, 112, 101, 114, 115, 105, 115, 116, 101, 110, 116, 47, 115, 95, 116, 111, 112, 105, 99, 45, 112, 97, 114, 116, 105, 116, 105, 111, 110, 45, 48], application=[112, 117, 108, 115, 97, 114]} - metadata ops timeout : 60 seconds
15:23:05.780 [main-EventThread] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/default/persistent/s_topic-partition-0] Created new ledger 974
15:23:05.900 [bookkeeper-ml-scheduler-OrderedScheduler-7-0] WARN org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - Cursor: ManagedCursorImpl{ledger=public/default/persistent/s_topic-partition-0, name=sub, ackPos=669:24781, readPos=669:24782} does not exist in the managed-ledger.
15:23:05.930 [bookkeeper-ml-scheduler-OrderedScheduler-7-0] WARN org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - Cursor: NonDurableCursorImpl{ledger=public/default/persistent/s_topic-partition-0, ackPos=669:-1, readPos=669:0} does not exist in the managed-ledger.
15:40:05.625 [BookKeeperClientWorker-OrderedExecutor-11-0] INFO org.apache.bookkeeper.mledger.impl.OpAddEntry - [public/default/persistent/s_topic-partition-0] Closing ledger 974 for being full
15:40:05.740 [main-EventThread] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/default/persistent/s_topic-partition-0] Creating a new ledger after closed
15:40:05.777 [BookKeeperClientWorker-OrderedExecutor-5-0] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/default/persistent/s_topic-partition-0] Creating ledger, metadata: {component=[109, 97, 110, 97, 103, 101, 100, 45, 108, 101, 100, 103, 101, 114], pulsar/managed-ledger=[112, 117, 98, 108, 105, 99, 47, 100, 101, 102, 97, 117, 108, 116, 47, 112, 101, 114, 115, 105, 115, 116, 101, 110, 116, 47, 115, 95, 116, 111, 112, 105, 99, 45, 112, 97, 114, 116, 105, 116, 105, 111, 110, 45, 48], application=[112, 117, 108, 115, 97, 114]} - metadata ops timeout : 60 seconds
15:40:06.049 [main-EventThread] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/default/persistent/s_topic-partition-0] Created new ledger 1017
15:40:06.151 [bookkeeper-ml-scheduler-OrderedScheduler-7-0] WARN org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - Cursor: ManagedCursorImpl{ledger=public/default/persistent/s_topic-partition-0, name=sub, ackPos=669:24781, readPos=669:24782} does not exist in the managed-ledger.
15:40:06.178 [bookkeeper-ml-scheduler-OrderedScheduler-7-0] WARN org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - Cursor: NonDurableCursorImpl{ledger=public/default/persistent/s_topic-partition-0, ackPos=669:-1, readPos=669:0} does not exist in the managed-ledger.
15:57:06.683 [BookKeeperClientWorker-OrderedExecutor-11-0] INFO org.apache.bookkeeper.mledger.impl.OpAddEntry - [public/default/persistent/s_topic-partition-0] Closing ledger 1017 for being full
15:57:06.800 [main-EventThread] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/default/persistent/s_topic-partition-0] Creating a new ledger after closed
15:57:06.824 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/default/persistent/s_topic-partition-0] Creating ledger, metadata: {component=[109, 97, 110, 97, 103, 101, 100, 45, 108, 101, 100, 103, 101, 114], pulsar/managed-ledger=[112, 117, 98, 108, 105, 99, 47, 100, 101, 102, 97, 117, 108, 116, 47, 112, 101, 114, 115, 105, 115, 116, 101, 110, 116, 47, 115, 95, 116, 111, 112, 105, 99, 45, 112, 97, 114, 116, 105, 116, 105, 111, 110, 45, 48], application=[112, 117, 108, 115, 97, 114]} - metadata ops timeout : 60 seconds
15:57:07.055 [main-EventThread] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/default/persistent/s_topic-partition-0] Created new ledger 1070
15:57:07.170 [bookkeeper-ml-scheduler-OrderedScheduler-7-0] WARN org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - Cursor: ManagedCursorImpl{ledger=public/default/persistent/s_topic-partition-0, name=sub, ackPos=669:24781, readPos=669:24782} does not exist in the managed-ledger.
15:57:07.194 [bookkeeper-ml-scheduler-OrderedScheduler-7-0] WARN org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - Cursor: NonDurableCursorImpl{ledger=public/default/persistent/s_topic-partition-0, ackPos=669:-1, readPos=669:0} does not exist in the managed-ledger.
16:14:07.387 [BookKeeperClientWorker-OrderedExecutor-11-0] INFO org.apache.bookkeeper.mledger.impl.OpAddEntry - [public/default/persistent/s_topic-partition-0] Closing ledger 1070 for being full
16:14:07.507 [main-EventThread] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/default/persistent/s_topic-partition-0] Creating a new ledger after closed
16:14:07.550 [BookKeeperClientWorker-OrderedExecutor-3-0] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/default/persistent/s_topic-partition-0] Creating ledger, metadata: {component=[109, 97, 110, 97, 103, 101, 100, 45, 108, 101, 100, 103, 101, 114], pulsar/managed-ledger=[112, 117, 98, 108, 105, 99, 47, 100, 101, 102, 97, 117, 108, 116, 47, 112, 101, 114, 115, 105, 115, 116, 101, 110, 116, 47, 115, 95, 116, 111, 112, 105, 99, 45, 112, 97, 114, 116, 105, 116, 105, 111, 110, 45, 48], application=[112, 117, 108, 115, 97, 114]} - metadata ops timeout : 60 seconds
16:14:07.798 [main-EventThread] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/default/persistent/s_topic-partition-0] Created new ledger 1097
16:14:07.917 [bookkeeper-ml-scheduler-OrderedScheduler-7-0] WARN org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - Cursor: ManagedCursorImpl{ledger=public/default/persistent/s_topic-partition-0, name=sub, ackPos=669:24781, readPos=669:24782} does not exist in the managed-ledger.
16:14:07.941 [bookkeeper-ml-scheduler-OrderedScheduler-7-0] WARN org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - Cursor: NonDurableCursorImpl{ledger=public/default/persistent/s_topic-partition-0, ackPos=669:-1, readPos=669:0} does not exist in the managed-ledger.
16:31:08.874 [BookKeeperClientWorker-OrderedExecutor-11-0] INFO org.apache.bookkeeper.mledger.impl.OpAddEntry - [public/default/persistent/s_topic-partition-0] Closing ledger 1097 for being full
16:31:08.991 [main-EventThread] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/default/persistent/s_topic-partition-0] Creating a new ledger after closed
16:31:09.017 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/default/persistent/s_topic-partition-0] Creating ledger, metadata: {component=[109, 97, 110, 97, 103, 101, 100, 45, 108, 101, 100, 103, 101, 114], pulsar/managed-ledger=[112, 117, 98, 108, 105, 99, 47, 100, 101, 102, 97, 117, 108, 116, 47, 112, 101, 114, 115, 105, 115, 116, 101, 110, 116, 47, 115, 95, 116, 111, 112, 105, 99, 45, 112, 97, 114, 116, 105, 116, 105, 111, 110, 45, 48], application=[112, 117, 108, 115, 97, 114]} - metadata ops timeout : 60 seconds
16:31:09.248 [main-EventThread] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/default/persistent/s_topic-partition-0] Created new ledger 1150
16:31:09.373 [bookkeeper-ml-scheduler-OrderedScheduler-7-0] WARN org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - Cursor: ManagedCursorImpl{ledger=public/default/persistent/s_topic-partition-0, name=sub, ackPos=669:24781, readPos=669:24782} does not exist in the managed-ledger.
16:31:09.398 [bookkeeper-ml-scheduler-OrderedScheduler-7-0] WARN org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - Cursor: NonDurableCursorImpl{ledger=public/default/persistent/s_topic-partition-0, ackPos=669:-1, readPos=669:0} does not exist in the managed-ledger.
16:48:07.475 [BookKeeperClientWorker-OrderedExecutor-11-0] INFO org.apache.bookkeeper.mledger.impl.OpAddEntry - [public/default/persistent/s_topic-partition-0] Closing ledger 1150 for being full
16:48:07.764 [BookKeeperClientWorker-OrderedExecutor-11-0] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/default/persistent/s_topic-partition-0] Creating a new ledger
16:48:07.788 [BookKeeperClientWorker-OrderedExecutor-11-0] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/default/persistent/s_topic-partition-0] Creating ledger, metadata: {component=[109, 97, 110, 97, 103, 101, 100, 45, 108, 101, 100, 103, 101, 114], pulsar/managed-ledger=[112, 117, 98, 108, 105, 99, 47, 100, 101, 102, 97, 117, 108, 116, 47, 112, 101, 114, 115, 105, 115, 116, 101, 110, 116, 47, 115, 95, 116, 111, 112, 105, 99, 45, 112, 97, 114, 116, 105, 116, 105, 111, 110, 45, 48], application=[112, 117, 108, 115, 97, 114]} - metadata ops timeout : 60 seconds
16:48:08.063 [main-EventThread] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/default/persistent/s_topic-partition-0] Created new ledger 1203
16:48:08.178 [bookkeeper-ml-scheduler-OrderedScheduler-7-0] WARN org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - Cursor: ManagedCursorImpl{ledger=public/default/persistent/s_topic-partition-0, name=sub, ackPos=669:24781, readPos=669:24782} does not exist in the managed-ledger.
16:48:08.201 [bookkeeper-ml-scheduler-OrderedScheduler-7-0] WARN org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - Cursor: NonDurableCursorImpl{ledger=public/default/persistent/s_topic-partition-0, ackPos=669:-1, readPos=669:0} does not exist in the managed-ledger.
16:54:55.202 [pulsar-web-49-18] INFO org.eclipse.jetty.server.RequestLog - 127.0.0.1 - - [26/Mar/2022:16:54:55 +0800] "GET /admin/v2/persistent/public/default/s_topic-partition-0/stats?getPreciseBacklog=false&subscriptionBacklogSize=false HTTP/1.1" 200 8996 "-" "Pulsar-Java-v2.9.2" 112
16:55:04.903 [pulsar-web-49-18] INFO org.eclipse.jetty.server.RequestLog - 127.0.0.1 - - [26/Mar/2022:16:55:04 +0800] "GET /admin/v2/persistent/public/default/s_topic-partition-0/stats?getPreciseBacklog=false&subscriptionBacklogSize=false HTTP/1.1" 200 8996 "-" "Pulsar-Java-v2.9.2" 19
16:55:50.078 [pulsar-web-49-18] INFO org.eclipse.jetty.server.RequestLog - 127.0.0.1 - - [26/Mar/2022:16:55:49 +0800] "GET /admin/v2/persistent/public/default/s_topic-partition-0/internalStats?metadata=false HTTP/1.1" 200 2442 "-" "Pulsar-Java-v2.9.2" 82
16:56:28.289 [pulsar-web-49-18] INFO org.eclipse.jetty.server.RequestLog - 127.0.0.1 - - [26/Mar/2022:16:56:28 +0800] "GET /admin/v3/transactions/transactionBufferStats/public/default/s_topic-partition-0 HTTP/1.1" 200 87 "-" "Pulsar-Java-v2.9.2" 161
16:56:38.145 [pulsar-web-49-18] INFO org.eclipse.jetty.server.RequestLog - 127.0.0.1 - - [26/Mar/2022:16:56:38 +0800] "GET /admin/v3/transactions/transactionBufferStats/public/default/s_topic-partition-0 HTTP/1.1" 200 87 "-" "Pulsar-Java-v2.9.2" 2
16:56:47.635 [pulsar-web-49-18] INFO org.eclipse.jetty.server.RequestLog - 127.0.0.1 - - [26/Mar/2022:16:56:47 +0800] "GET /admin/v3/transactions/transactionBufferStats/public/default/s_topic-partition-0 HTTP/1.1" 200 87 "-" "Pulsar-Java-v2.9.2" 2
16:57:09.046 [pulsar-web-49-18] INFO org.eclipse.jetty.server.RequestLog - 127.0.0.1 - - [26/Mar/2022:16:57:09 +0800] "GET /admin/v2/persistent/public/default/s_topic-partition-0/stats?getPreciseBacklog=false&subscriptionBacklogSize=false HTTP/1.1" 200 8998 "-" "Pulsar-Java-v2.9.2" 3
16:58:43.952 [pulsar-web-49-18] INFO org.eclipse.jetty.server.RequestLog - 127.0.0.1 - - [26/Mar/2022:16:58:43 +0800] "GET /lookup/v2/topic/persistent/public/default/s_topic-partition-0 HTTP/1.1" 200 129 "-" "Pulsar-Java-v2.9.2" 88
```
The broker jstack, looks like don't have any dead lock
[broker-0.jstack.log](https://github.com/apache/pulsar/files/8355081/broker-0.jstack.log)
**Additional context**
https://github.com/apache/pulsar/commit/f0a2171cbad894cec5bfb2d4de31cb8de32a3183
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mlyahmed commented on issue #14889: All the consumers get stuck for a broker after enabled transaction
Posted by GitBox <gi...@apache.org>.
mlyahmed commented on issue #14889:
URL: https://github.com/apache/pulsar/issues/14889#issuecomment-1081106264
Hi @codelipenghui, I would like to take it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org