You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@rocketmq.apache.org by GitBox <gi...@apache.org> on 2020/12/30 06:33:18 UTC
[GitHub] [rocketmq-client-cpp] zhangyulu opened a new issue #293: consumer stuck in rebalance and doesn't consume messsages
zhangyulu opened a new issue #293:
URL: https://github.com/apache/rocketmq-client-cpp/issues/293
1. Please describe the issue you observed:
- What did you do (The steps to reproduce)?
let consumer run
- What did you expect to see?
keep consuming messages
- What did you see instead?
stopped consuming messages
2. Please tell us about your environment:
- What is your OS?
CentOS Linux 7.6.1810
- What is your client version?
commit hash: b29bf5b69eab187a813a4d7f59534a6375e32d25
commited at 2020.03.23
- What is your RocketMQ version?
V4_3_0
3. Other information (e.g. detailed explanation, logs, related issues, suggestions on how to fix, etc):
rocketmq log:
[2020-Mar-28 08:42:42.124297](info):unlock success of mq:MessageQueue [topic=browser, brokerName=broker-1, queueId=1][unlock:335]
[2020-Mar-28 08:42:42.124426](warning):PullRequest for [MessageQueue [topic=browser, brokerName=broker-1, queueId=2]] has been expired 9111 ms,m_lastPullTimestamp = 1585356033013 ms[isPullRequestExpired:211]
[2020-Mar-28 08:42:42.124486](info):Drop mq:MessageQueue [topic=browser, brokerName=broker-1, queueId=2] according Pull timeout.[updateRequestTableInRebalance:501]
[2020-Mar-28 08:42:42.124554](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=browser, brokerName=broker-1, queueId=2], its offset is:268305037[updateConsumeOffsetToBroker:310]
日志一直循环出现上述内容。
只出现在跨机房消费的场景（ping延迟40ms左右），网络可能会有波动但从未断开。每次消费者启动后可以正常运行一段时间（几个小时或者几天），但在某个时间点突然不再消费新消息，导致该消费者所分配到的队列堆积，也不会再分配给其他正常消费者。超时设置(try lock timeout & connect timeout)均使用默认值。
共部署4-6个消费者，任意一个消费者均多次出现过上述情况，必须重启消费者才能恢复。
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq-client-cpp] YellowOldOdd commented on issue #293: consumer stuck in rebalance and doesn't consume messsages
Posted by GitBox <gi...@apache.org>.
YellowOldOdd commented on issue #293:
URL: https://github.com/apache/rocketmq-client-cpp/issues/293#issuecomment-1067903481
嗨 @gaoyf ， 请问用了#[5629226997fea8bc6ac91e9481d2140d983106a7](https://github.com/apache/rocketmq-client-cpp/commit/5629226997fea8bc6ac91e9481d2140d983106a7)的改动后，问题有解决吗？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq-client-cpp] gaoyf commented on issue #293: consumer stuck in rebalance and doesn't consume messsages
Posted by GitBox <gi...@apache.org>.
gaoyf commented on issue #293:
URL: https://github.com/apache/rocketmq-client-cpp/issues/293#issuecomment-1015021342
> > 我也是，日志也一样，@YellowOldOdd @ifplusor @ShannonDing
>
> 你是什么消费模式？
就是PushConsumer，跟@YellowOldOdd的一模一样，我看master代码已经修改了 #[5629226997fea8bc6ac91e9481d2140d983106a7](https://github.com/apache/rocketmq-client-cpp/commit/5629226997fea8bc6ac91e9481d2140d983106a7)，应该就是这个问题，能不能release一个2.2.1呢，这个问题太严重了
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq-client-cpp] gaoyf commented on issue #293: consumer stuck in rebalance and doesn't consume messsages
Posted by GitBox <gi...@apache.org>.
gaoyf commented on issue #293:
URL: https://github.com/apache/rocketmq-client-cpp/issues/293#issuecomment-1014326087
我也是，日志也一样，@YellowOldOdd @ifplusor @ShannonDing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq-client-cpp] zhangyulu commented on issue #293: consumer stuck in rebalance and doesn't consume messsages
Posted by GitBox <gi...@apache.org>.
zhangyulu commented on issue #293:
URL: https://github.com/apache/rocketmq-client-cpp/issues/293#issuecomment-763578353
[stack.log](https://github.com/apache/rocketmq-client-cpp/files/5842403/stack.log)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq-client-cpp] ifplusor commented on issue #293: consumer stuck in rebalance and doesn't consume messsages
Posted by GitBox <gi...@apache.org>.
ifplusor commented on issue #293:
URL: https://github.com/apache/rocketmq-client-cpp/issues/293#issuecomment-764389632
@zhangyulu 我做了一些测试，但没办法复现你的遇到的问题。在你的日志里，负载均衡、mq加锁、offset上报都是正常的。你可以先检查一下参数设置和consumer group的积压情况，并在```DefaultMQPushConsumerImpl::executePullRequestImmediately()```函数里加一些日志，验证消息拉取是否正常。
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq-client-cpp] zhangyulu commented on issue #293: consumer stuck in rebalance and doesn't consume messsages
Posted by GitBox <gi...@apache.org>.
zhangyulu commented on issue #293:
URL: https://github.com/apache/rocketmq-client-cpp/issues/293#issuecomment-763306409
re_dev分支稳定性如何？另外昨天试了一下，部分api不兼容(如setSessionCredentials)，需要修改代码
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq-client-cpp] gaoyf edited a comment on issue #293: consumer stuck in rebalance and doesn't consume messsages
Posted by GitBox <gi...@apache.org>.
gaoyf edited a comment on issue #293:
URL: https://github.com/apache/rocketmq-client-cpp/issues/293#issuecomment-1015021342
> > 我也是，日志也一样，@YellowOldOdd @ifplusor @ShannonDing
>
> 你是什么消费模式？
就是PushConsumer，我一共有4个broker，每个broker8个队列，启动了30个消费者，这个问题必现。跟@YellowOldOdd的一模一样，我看master代码已经修改了 #[5629226997fea8bc6ac91e9481d2140d983106a7](https://github.com/apache/rocketmq-client-cpp/commit/5629226997fea8bc6ac91e9481d2140d983106a7)，应该就是这个问题，能不能release一个2.2.1呢，这个问题太严重了
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq-client-cpp] ifplusor commented on issue #293: consumer stuck in rebalance and doesn't consume messsages
Posted by GitBox <gi...@apache.org>.
ifplusor commented on issue #293:
URL: https://github.com/apache/rocketmq-client-cpp/issues/293#issuecomment-752364984
@zhangyulu master分支的消费管理有点乱，你能把你当前使用的源码库发出来吗？或者，你也可以尝试切换到rd_dev分支。
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq-client-cpp] zhangyulu commented on issue #293: consumer stuck in rebalance and doesn't consume messsages
Posted by GitBox <gi...@apache.org>.
zhangyulu commented on issue #293:
URL: https://github.com/apache/rocketmq-client-cpp/issues/293#issuecomment-763291780
@ShannonDing 能否帮忙看一下这个问题呢
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq-client-cpp] ifplusor commented on issue #293: consumer stuck in rebalance and doesn't consume messsages
Posted by GitBox <gi...@apache.org>.
ifplusor commented on issue #293:
URL: https://github.com/apache/rocketmq-client-cpp/issues/293#issuecomment-763312936
> re_dev分支稳定性如何？另外昨天试了一下，部分api不兼容(如setSessionCredentials)，需要修改代码
比master稳定很多。re_dev的api重新设计了，所以不兼容。但后面可能还会加一层专门用于向后兼容的wrapper，社区正在评估。
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq-client-cpp] zhangyulu commented on issue #293: consumer stuck in rebalance and doesn't consume messsages
Posted by GitBox <gi...@apache.org>.
zhangyulu commented on issue #293:
URL: https://github.com/apache/rocketmq-client-cpp/issues/293#issuecomment-763527619
合并更改之后还是无法消费
[tmp.log](https://github.com/apache/rocketmq-client-cpp/files/5841922/tmp.log)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq-client-cpp] ifplusor commented on issue #293: consumer stuck in rebalance and doesn't consume messsages
Posted by GitBox <gi...@apache.org>.
ifplusor commented on issue #293:
URL: https://github.com/apache/rocketmq-client-cpp/issues/293#issuecomment-763440482
@zhangyulu RebalanceImpl::lock() 方法有bug，我已经改了。
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq-client-cpp] YellowOldOdd commented on issue #293: consumer stuck in rebalance and doesn't consume messsages
Posted by GitBox <gi...@apache.org>.
YellowOldOdd commented on issue #293:
URL: https://github.com/apache/rocketmq-client-cpp/issues/293#issuecomment-871354343
这是卡住时附近的日志
这是卡住时的日志:
[2021-Jun-30 11:17:15.192174](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], its offset is:7971061[updateConsumeOffsetToBroker:310]
[2021-Jun-30 11:17:20.194643](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], its offset is:7971062[updateConsumeOffsetToBroker:310]
[2021-Jun-30 11:17:25.192164](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], its offset is:7971063[updateConsumeOffsetToBroker:310]
[2021-Jun-30 11:17:30.194986](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], its offset is:7971068[updateConsumeOffsetToBroker:310]
[2021-Jun-30 11:17:35.192150](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], its offset is:7971070[updateConsumeOffsetToBroker:310]
[2021-Jun-30 11:17:40.194668](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], its offset is:7971071[updateConsumeOffsetToBroker:310]
[2021-Jun-30 11:17:45.192193](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], its offset is:7971073[updateConsumeOffsetToBroker:310]
[2021-Jun-30 11:17:50.194961](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], its offset is:7971074[updateConsumeOffsetToBroker:310]
[2021-Jun-30 11:17:55.192106](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], its offset is:7971078[updateConsumeOffsetToBroker:310]
[2021-Jun-30 11:18:00.194443](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], its offset is:7971080[updateConsumeOffsetToBroker:310]
[2021-Jun-30 11:18:05.192152](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], its offset is:7971082[updateConsumeOffsetToBroker:310]
[2021-Jun-30 11:18:10.194912](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], its offset is:7971084[updateConsumeOffsetToBroker:310]
[2021-Jun-30 11:18:15.192283](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], its offset is:7971087[updateConsumeOffsetToBroker:310]
[2021-Jun-30 11:18:20.194508](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], its offset is:7971087[updateConsumeOffsetToBroker:310]
[2021-Jun-30 11:18:25.192197](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], its offset is:7971088[updateConsumeOffsetToBroker:310]
MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3]
MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3]
MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3]
[2021-Jun-30 11:18:30.194282](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], its offset is:7971090[updateConsumeOffsetToBroker:310]
[2021-Jun-30 11:18:30.891084](info):Drop mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3],because not responsive[updateRequestTableInRebalance:493]
[2021-Jun-30 11:18:30.891111](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], its offset is:7971091[updateConsumeOffsetToBroker:310]
[2021-Jun-30 11:18:30.932469](info):CONSUME_FROM_LAST_OFFSET, lastOffset of mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3] is:7971091[computePullFromWhere:568]
[2021-Jun-30 11:18:30.932487](info):Add mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], request initial offset:7971091[updateRequestTableInRebalance:528]
[2021-Jun-30 11:18:30.932550](info):Start to pull MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], offset:7971091, GroupName note-change-media-v2-consumer[updateRequestTableInRebalance:539]
MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3]
[2021-Jun-30 11:18:30.939869](info):Drop mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3],because not responsive[updateRequestTableInRebalance:493]
[2021-Jun-30 11:18:30.939894](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], its offset is:7971091[updateConsumeOffsetToBroker:310]
[2021-Jun-30 11:18:31.384323](info):CONSUME_FROM_LAST_OFFSET, lastOffset of mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3] is:7971091[computePullFromWhere:568]
[2021-Jun-30 11:18:31.384341](info):Add mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], request initial offset:7971091[updateRequestTableInRebalance:528]
[2021-Jun-30 11:18:31.384408](info):Start to pull MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], offset:7971091, GroupName note-change-media-v2-consumer[updateRequestTableInRebalance:539]
MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3]
[2021-Jun-30 11:18:31.401570](info):Drop mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3],because not responsive[updateRequestTableInRebalance:493]
[2021-Jun-30 11:18:31.401596](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], its offset is:7971091[updateConsumeOffsetToBroker:310]
[2021-Jun-30 11:18:31.422630](info):CONSUME_FROM_LAST_OFFSET, lastOffset of mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3] is:7971091[computePullFromWhere:568]
[2021-Jun-30 11:18:31.422653](info):Add mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], request initial offset:7971091[updateRequestTableInRebalance:528]
[2021-Jun-30 11:18:31.422711](info):Start to pull MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], offset:7971091, GroupName note-change-media-v2-consumer[updateRequestTableInRebalance:539]
MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3]
[2021-Jun-30 11:18:31.561148](info):Pull request for queue[MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3]] has been set as dropped. Will NOT pull this queue any more[onSuccess:58]
[2021-Jun-30 11:18:31.561140](info):Pull request for queue[MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3]] has been set as dropped. Will NOT pull this queue any more[onSuccess:58]
[2021-Jun-30 11:18:31.561183](info):Pull request for queue[MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3]] has been set as dropped. Will NOT pull this queue any more[onSuccess:58]
[2021-Jun-30 11:18:31.561214](info):Pull request for queue[MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3]] has been set as dropped. Will NOT pull this queue any more[onSuccess:58]
[2021-Jun-30 11:18:35.191912](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], its offset is:7971091[updateConsumeOffsetToBroker:310]
MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3]
MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3]
MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3]
MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3]
MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3]
[2021-Jun-30 11:18:40.194706](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], its offset is:7971091[updateConsumeOffsetToBroker:310]
[2021-Jun-30 11:18:45.192151](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], its offset is:7971091[updateConsumeOffsetToBroker:310]
[2021-Jun-30 11:18:50.194279](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], its offset is:7971091[updateConsumeOffsetToBroker:310]
[2021-Jun-30 11:18:55.192147](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], its offset is:7971091[updateConsumeOffsetToBroker:310]
[2021-Jun-30 11:19:00.194521](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], its offset is:7971091[updateConsumeOffsetToBroker:310]
[2021-Jun-30 11:19:05.192131](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], its offset is:7971091[updateConsumeOffsetToBroker:310]
[2021-Jun-30 11:19:10.194310](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], its offset is:7971091[updateConsumeOffsetToBroker:310]
[2021-Jun-30 11:19:15.192145](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], its offset is:7971091[updateConsumeOffsetToBroker:310]
[2021-Jun-30 11:19:20.194992](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], its offset is:7971091[updateConsumeOffsetToBroker:310]
[2021-Jun-30 11:19:25.192140](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], its offset is:7971091[updateConsumeOffsetToBroker:310]
[2021-Jun-30 11:19:30.194609](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], its offset is:7971091[updateConsumeOffsetToBroker:310]
[2021-Jun-30 11:19:35.192117](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], its offset is:7971091[updateConsumeOffsetToBroker:310]
[2021-Jun-30 11:19:40.194564](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], its offset is:7971091[updateConsumeOffsetToBroker:310]
[2021-Jun-30 11:19:45.192149](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], its offset is:7971091[updateConsumeOffsetToBroker:310]
[2021-Jun-30 11:19:50.194340](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], its offset is:7971091[updateConsumeOffsetToBroker:310]
[2021-Jun-30 11:19:55.201273](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], its offset is:7971091[updateConsumeOffsetToBroker:310]
[2021-Jun-30 11:20:00.194615](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], its offset is:7971091[updateConsumeOffsetToBroker:310]
[2021-Jun-30 11:20:05.192149](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], its offset is:7971091[updateConsumeOffsetToBroker:310]
[2021-Jun-30 11:20:10.194523](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], its offset is:7971091[updateConsumeOffsetToBroker:310]
[2021-Jun-30 11:20:15.192154](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], its offset is:7971091[updateConsumeOffsetToBroker:310]
[2021-Jun-30 11:20:20.194413](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], its offset is:7971091[updateConsumeOffsetToBroker:310]
[2021-Jun-30 11:20:25.192116](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], its offset is:7971091[updateConsumeOffsetToBroker:310]
[2021-Jun-30 11:20:30.194844](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], its offset is:7971091[updateConsumeOffsetToBroker:310]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq-client-cpp] ifplusor commented on issue #293: consumer stuck in rebalance and doesn't consume messsages
Posted by GitBox <gi...@apache.org>.
ifplusor commented on issue #293:
URL: https://github.com/apache/rocketmq-client-cpp/issues/293#issuecomment-764389632
@zhangyulu 我做了一些测试，但没办法复现你的遇到的问题。在你的日志里，负载均衡、mq加锁、offset上报都是正常的。你可以先检查一下参数设置和consumer group的积压情况，并在```DefaultMQPushConsumerImpl::executePullRequestImmediately()```函数里加一些日志，验证消息拉取是否正常。
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq-client-cpp] ifplusor closed issue #293: consumer stuck in rebalance and doesn't consume messsages
Posted by GitBox <gi...@apache.org>.
ifplusor closed issue #293:
URL: https://github.com/apache/rocketmq-client-cpp/issues/293
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq-client-cpp] ifplusor commented on issue #293: consumer stuck in rebalance and doesn't consume messsages
Posted by GitBox <gi...@apache.org>.
ifplusor commented on issue #293:
URL: https://github.com/apache/rocketmq-client-cpp/issues/293#issuecomment-1014899306
> 我也是，日志也一样，@YellowOldOdd @ifplusor @ShannonDing
你是什么消费模式？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq-client-cpp] ifplusor commented on issue #293: consumer stuck in rebalance and doesn't consume messsages
Posted by GitBox <gi...@apache.org>.
ifplusor commented on issue #293:
URL: https://github.com/apache/rocketmq-client-cpp/issues/293#issuecomment-752347657
version: commit b29bf5b69eab187a813a4d7f59534a6375e32d25 + patch #300
consume model: push-orderly
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq-client-cpp] zhangyulu commented on issue #293: consumer stuck in rebalance and doesn't consume messsages
Posted by GitBox <gi...@apache.org>.
zhangyulu commented on issue #293:
URL: https://github.com/apache/rocketmq-client-cpp/issues/293#issuecomment-752369257
> @zhangyulu master分支的消费管理有点乱，你能把你当前使用的源码库发出来吗？或者，你也可以尝试切换到rd_dev分支。
@ifplusor
[rocketmq-client-cpp-debug.tar.gz](https://github.com/apache/rocketmq-client-cpp/files/5753471/rocketmq-client-cpp-debug.tar.gz)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq-client-cpp] zhangyulu commented on issue #293: consumer stuck in rebalance and doesn't consume messsages
Posted by GitBox <gi...@apache.org>.
zhangyulu commented on issue #293:
URL: https://github.com/apache/rocketmq-client-cpp/issues/293#issuecomment-752312388
在线上又复现了若干次。用的版本是基于commit #b29bf5b69eab187a813a4d7f59534a6375e32d25，之后把PR #300中的变更手动合并进来，日志如下：
[389_rocketmq-cpp.log](https://github.com/apache/rocketmq-client-cpp/files/5752886/389_rocketmq-cpp.log)
在03:59:03时出现"wait response timeout of heartbeat"
之后的几个小时里，都会打出
oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=topic_xxx, brokerName=broker-Cluster-kseILdRbPSF-0, queueId=0], its offset is:5255956
这个offset一直没有变化，消费者也一直无法拉取到消息进行消费，直到重启消费者进程才恢复
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq-client-cpp] YellowOldOdd commented on issue #293: consumer stuck in rebalance and doesn't consume messsages
Posted by GitBox <gi...@apache.org>.
YellowOldOdd commented on issue #293:
URL: https://github.com/apache/rocketmq-client-cpp/issues/293#issuecomment-871354089
嗨，你好，我也遇到了一摸一样的问题，版本2.2.0，consumer数量发生变化时，PushConsumer在Rebalance时部分queue卡住
@ifplusor @ShannonDing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq-client-cpp] ifplusor commented on issue #293: consumer stuck in rebalance and doesn't consume messsages
Posted by GitBox <gi...@apache.org>.
ifplusor commented on issue #293:
URL: https://github.com/apache/rocketmq-client-cpp/issues/293#issuecomment-763546044
@zhouxinyu 调用下面这个命令看一下线程栈，PID是你的进程号
```bash
gdb -ex "thread apply all bt" -batch -p $PID
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq-client-cpp] zhangyulu edited a comment on issue #293: consumer stuck in rebalance and doesn't consume messsages
Posted by GitBox <gi...@apache.org>.
zhangyulu edited a comment on issue #293:
URL: https://github.com/apache/rocketmq-client-cpp/issues/293#issuecomment-763403268
刚刚试了一下，用re_dev分支编译出来example里面的OrderlyConsumer去消费一个topic，一直拉不到消息，但使用PushConsumer就能够正常消费
[8348_rocketmq-cpp.log](https://github.com/apache/rocketmq-client-cpp/files/5840847/8348_rocketmq-cpp.log)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq-client-cpp] YellowOldOdd commented on issue #293: consumer stuck in rebalance and doesn't consume messsages
Posted by GitBox <gi...@apache.org>.
YellowOldOdd commented on issue #293:
URL: https://github.com/apache/rocketmq-client-cpp/issues/293#issuecomment-871354743
基本可以稳定复现，请问有什么推荐的解决方法吗？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq-client-cpp] ifplusor commented on issue #293: consumer stuck in rebalance and doesn't consume messsages
Posted by GitBox <gi...@apache.org>.
ifplusor commented on issue #293:
URL: https://github.com/apache/rocketmq-client-cpp/issues/293#issuecomment-763297304
@zhangyulu 你这个问题我之前看了，从已有的日志里没有分析出问题的原因，建议换 re_dev 分支的版本看看效果。
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq-client-cpp] YellowOldOdd edited a comment on issue #293: consumer stuck in rebalance and doesn't consume messsages
Posted by GitBox <gi...@apache.org>.
YellowOldOdd edited a comment on issue #293:
URL: https://github.com/apache/rocketmq-client-cpp/issues/293#issuecomment-871354343
这是某个queue被卡住时的日志
[2021-Jun-30 11:18:00.194443](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], its offset is:7971080[updateConsumeOffsetToBroker:310]
[2021-Jun-30 11:18:05.192152](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], its offset is:7971082[updateConsumeOffsetToBroker:310]
[2021-Jun-30 11:18:10.194912](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], its offset is:7971084[updateConsumeOffsetToBroker:310]
[2021-Jun-30 11:18:15.192283](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], its offset is:7971087[updateConsumeOffsetToBroker:310]
[2021-Jun-30 11:18:20.194508](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], its offset is:7971087[updateConsumeOffsetToBroker:310]
[2021-Jun-30 11:18:25.192197](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], its offset is:7971088[updateConsumeOffsetToBroker:310]
MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3]
MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3]
MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3]
[2021-Jun-30 11:18:30.194282](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], its offset is:7971090[updateConsumeOffsetToBroker:310]
[2021-Jun-30 11:18:30.891084](info):Drop mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3],because not responsive[updateRequestTableInRebalance:493]
[2021-Jun-30 11:18:30.891111](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], its offset is:7971091[updateConsumeOffsetToBroker:310]
[2021-Jun-30 11:18:30.932469](info):CONSUME_FROM_LAST_OFFSET, lastOffset of mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3] is:7971091[computePullFromWhere:568]
[2021-Jun-30 11:18:30.932487](info):Add mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], request initial offset:7971091[updateRequestTableInRebalance:528]
[2021-Jun-30 11:18:30.932550](info):Start to pull MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], offset:7971091, GroupName note-change-media-v2-consumer[updateRequestTableInRebalance:539]
MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3]
[2021-Jun-30 11:18:30.939869](info):Drop mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3],because not responsive[updateRequestTableInRebalance:493]
[2021-Jun-30 11:18:30.939894](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], its offset is:7971091[updateConsumeOffsetToBroker:310]
[2021-Jun-30 11:18:31.384323](info):CONSUME_FROM_LAST_OFFSET, lastOffset of mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3] is:7971091[computePullFromWhere:568]
[2021-Jun-30 11:18:31.384341](info):Add mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], request initial offset:7971091[updateRequestTableInRebalance:528]
[2021-Jun-30 11:18:31.384408](info):Start to pull MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], offset:7971091, GroupName note-change-media-v2-consumer[updateRequestTableInRebalance:539]
MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3]
[2021-Jun-30 11:18:31.401570](info):Drop mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3],because not responsive[updateRequestTableInRebalance:493]
[2021-Jun-30 11:18:31.401596](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], its offset is:7971091[updateConsumeOffsetToBroker:310]
[2021-Jun-30 11:18:31.422630](info):CONSUME_FROM_LAST_OFFSET, lastOffset of mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3] is:7971091[computePullFromWhere:568]
[2021-Jun-30 11:18:31.422653](info):Add mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], request initial offset:7971091[updateRequestTableInRebalance:528]
[2021-Jun-30 11:18:31.422711](info):Start to pull MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], offset:7971091, GroupName note-change-media-v2-consumer[updateRequestTableInRebalance:539]
MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3]
[2021-Jun-30 11:18:31.561148](info):Pull request for queue[MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3]] has been set as dropped. Will NOT pull this queue any more[onSuccess:58]
[2021-Jun-30 11:18:31.561140](info):Pull request for queue[MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3]] has been set as dropped. Will NOT pull this queue any more[onSuccess:58]
[2021-Jun-30 11:18:31.561183](info):Pull request for queue[MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3]] has been set as dropped. Will NOT pull this queue any more[onSuccess:58]
[2021-Jun-30 11:18:31.561214](info):Pull request for queue[MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3]] has been set as dropped. Will NOT pull this queue any more[onSuccess:58]
[2021-Jun-30 11:18:35.191912](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], its offset is:7971091[updateConsumeOffsetToBroker:310]
MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3]
MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3]
MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3]
MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3]
MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3]
[2021-Jun-30 11:18:40.194706](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], its offset is:7971091[updateConsumeOffsetToBroker:310]
[2021-Jun-30 11:18:45.192151](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], its offset is:7971091[updateConsumeOffsetToBroker:310]
[2021-Jun-30 11:18:50.194279](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], its offset is:7971091[updateConsumeOffsetToBroker:310]
[2021-Jun-30 11:18:55.192147](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3], its offset is:7971091[updateConsumeOffsetToBroker:310]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq-client-cpp] ifplusor commented on issue #293: consumer stuck in rebalance and doesn't consume messsages
Posted by GitBox <gi...@apache.org>.
ifplusor commented on issue #293:
URL: https://github.com/apache/rocketmq-client-cpp/issues/293#issuecomment-1014898795
> 嗨，你好，我也遇到了一摸一样的问题，版本2.2.0，consumer数量发生变化时，PushConsumer在Rebalance时部分queue卡住
>
> @ifplusor @ShannonDing
你是什么消费模式？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq-client-cpp] ifplusor commented on issue #293: consumer stuck in rebalance and doesn't consume messsages
Posted by GitBox <gi...@apache.org>.
ifplusor commented on issue #293:
URL: https://github.com/apache/rocketmq-client-cpp/issues/293#issuecomment-1015028659
@ShannonDing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq-client-cpp] zhangyulu commented on issue #293: consumer stuck in rebalance and doesn't consume messsages
Posted by GitBox <gi...@apache.org>.
zhangyulu commented on issue #293:
URL: https://github.com/apache/rocketmq-client-cpp/issues/293#issuecomment-763403268
刚刚试了一下，用re_dev分支编译出来example里面的OrderlyConsumer去消费一个topic，一直没法拉取消息，但使用PushConsumer就能够正常消费
[8348_rocketmq-cpp.log](https://github.com/apache/rocketmq-client-cpp/files/5840847/8348_rocketmq-cpp.log)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq-client-cpp] YellowOldOdd commented on issue #293: consumer stuck in rebalance and doesn't consume messsages
Posted by GitBox <gi...@apache.org>.
YellowOldOdd commented on issue #293:
URL: https://github.com/apache/rocketmq-client-cpp/issues/293#issuecomment-871377581
然后，"任意一个消费者均多次出现过上述情况，必须重启消费者才能恢复"。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq-client-cpp] YellowOldOdd commented on issue #293: consumer stuck in rebalance and doesn't consume messsages
Posted by GitBox <gi...@apache.org>.
YellowOldOdd commented on issue #293:
URL: https://github.com/apache/rocketmq-client-cpp/issues/293#issuecomment-871357856
并且在接下来的几个小时里，会一直报
[2021-Jun-30 14:31:20.195386](warning):PullRequest for [MessageQueue [topic=note-change, brokerName=cluster02-broker10, queueId=3]] has been expired 9987 ms,m_lastPullTimestamp = 1625034550208 ms[isPullRequestExpired:211]
的超时错误，然后逐渐恢复了。。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org