You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@rocketmq.apache.org by GitBox <gi...@apache.org> on 2022/08/18 07:42:45 UTC
[GitHub] [rocketmq] MrLYG opened a new issue, #4840: HA模式下master宕机无法从slave节点消费；In HA mode, the master is down and cannot be consumed from the slave
MrLYG opened a new issue, #4840:
URL: https://github.com/apache/rocketmq/issues/4840
**ISSUE REPORT**
1. Please describe the issue you observed:
- What did you do (The steps to reproduce)?
step1. K8S使用operator部署HA模式集群1master&1slave
step1. K8S uses operator to deploy HA mode cluster 1master&1slave
step2. 往集群名为slave-test的topic写入一些数据
step2. Write some data to the topic of the cluster named slave-test
step3.模拟master宕机，关闭master节点
step3. Simulate the master downtime and shut down the master
step4. 启动消费者服务，消费数据该topic数据
step4. Start the consumer service and consume data for the topic data
- What is expected to see?
在master宕机的情况下，slave应当承担数据消费职责，不会影响rocketmq集群内topic的消息消费任务。
In the case of master downtime, the slave should assume the responsibility of data consumption and will not affect the message consumption task of topics in the rocketmq cluster.
- What did you see instead?
消费者消费消息失败；
Consumer failed to consume message
```C:\Users\liyuangang\AppData\Local\Temp\GoLand\___9go_build_client.exe
nameserverF =
topicF = SELF_TEST_TOPIC
consumerGroupF = slave-test
producerGroupF = slave-test
time="2022-08-18T15:37:54+08:00" level=info msg="the consumer start beginning" consumerGroup=slave-test messageModel=Clustering unitMode=false
time="2022-08-18T15:37:54+08:00" level=info msg="the topic route info changed" changeTo="{\"OrderTopicConf\":\"\",\"queueDatas\":[{\"brokerName\":\"rocketmq-broker-0\",\"readQueueNums\":4,\"writeQueueNums\":4,\"perm\":6,\"topicSy
nFlag\":0}],\"brokerDatas\":[{\"cluster\":\"rocketmq-broker\",\"brokerName\":\"rocketmq-broker-0\",\"brokerAddrs\":{\"1\":\"10.53.70.232:23333\"}}]}" changedFrom="<nil>" topic=slave-test
time="2022-08-18T15:37:54+08:00" level=warning msg="query topic route from server error" underlayError="topic not exist"
time="2022-08-18T15:37:54+08:00" level=warning msg="queryTopicRouteInfoFromServer return nil" topic="%RETRY%slave-test"
time="2022-08-18T15:37:54+08:00" level=warning msg="query topic route from server error" underlayError="topic not exist"
time="2022-08-18T15:37:54+08:00" level=warning msg="queryTopicRouteInfoFromServer return nil" topic="%RETRY%slave-test"
time="2022-08-18T15:37:55+08:00" level=info msg="receive broker's notification to consumer group" consumerGroup=slave-test
time="2022-08-18T15:37:55+08:00" level=warning msg="do balance in group failed, the topic does not exist" consumerGroup=slave-test topic="%RETRY%slave-test"
time="2022-08-18T15:37:55+08:00" level=warning msg="delete mq from offset table" MessageQueue="MessageQueue [topic=slave-test, brokerName=rocketmq-broker-0, queueId=3]" consumerGroup=slave-test
time="2022-08-18T15:37:55+08:00" level=error msg="fecth offset of mq from broker error" MessageQueue="MessageQueue [topic=slave-test, brokerName=rocketmq-broker-0, queueId=3]" consumerGroup=slave-test underlayError="broker: rocke
tmq-broker-0 address not found"
time="2022-08-18T15:37:55+08:00" level=warning msg="query max offset error" MessageQueue="MessageQueue [topic=slave-test, brokerName=rocketmq-broker-0, queueId=3]" underlayError="the broker [rocketmq-broker-0] does not exist"
time="2022-08-18T15:37:55+08:00" level=warning msg="do defaultConsumer, add a new mq failed" MessageQueue="MessageQueue [topic=slave-test, brokerName=rocketmq-broker-0, queueId=3]" consumerGroup=slave-test
time="2022-08-18T15:37:55+08:00" level=warning msg="delete mq from offset table" MessageQueue="MessageQueue [topic=slave-test, brokerName=rocketmq-broker-0, queueId=0]" consumerGroup=slave-test
time="2022-08-18T15:37:55+08:00" level=error msg="fecth offset of mq from broker error" MessageQueue="MessageQueue [topic=slave-test, brokerName=rocketmq-broker-0, queueId=0]" consumerGroup=slave-test underlayError="broker: rocke
tmq-broker-0 address not found"
time="2022-08-18T15:37:55+08:00" level=warning msg="query max offset error" MessageQueue="MessageQueue [topic=slave-test, brokerName=rocketmq-broker-0, queueId=0]" underlayError="the broker [rocketmq-broker-0] does not exist"
time="2022-08-18T15:37:55+08:00" level=warning msg="do defaultConsumer, add a new mq failed" MessageQueue="MessageQueue [topic=slave-test, brokerName=rocketmq-broker-0, queueId=0]" consumerGroup=slave-test
time="2022-08-18T15:37:55+08:00" level=warning msg="delete mq from offset table" MessageQueue="MessageQueue [topic=slave-test, brokerName=rocketmq-broker-0, queueId=1]" consumerGroup=slave-test
time="2022-08-18T15:37:55+08:00" level=error msg="fecth offset of mq from broker error" MessageQueue="MessageQueue [topic=slave-test, brokerName=rocketmq-broker-0, queueId=1]" consumerGroup=slave-test underlayError="broker: rocke
tmq-broker-0 address not found"
time="2022-08-18T15:37:55+08:00" level=warning msg="query max offset error" MessageQueue="MessageQueue [topic=slave-test, brokerName=rocketmq-broker-0, queueId=1]" underlayError="the broker [rocketmq-broker-0] does not exist"
time="2022-08-18T15:37:55+08:00" level=warning msg="do defaultConsumer, add a new mq failed" MessageQueue="MessageQueue [topic=slave-test, brokerName=rocketmq-broker-0, queueId=1]" consumerGroup=slave-test
time="2022-08-18T15:37:55+08:00" level=warning msg="delete mq from offset table" MessageQueue="MessageQueue [topic=slave-test, brokerName=rocketmq-broker-0, queueId=2]" consumerGroup=slave-test
time="2022-08-18T15:37:55+08:00" level=error msg="fecth offset of mq from broker error" MessageQueue="MessageQueue [topic=slave-test, brokerName=rocketmq-broker-0, queueId=2]" consumerGroup=slave-test underlayError="broker: rocke
tmq-broker-0 address not found"
time="2022-08-18T15:37:55+08:00" level=warning msg="query max offset error" MessageQueue="MessageQueue [topic=slave-test, brokerName=rocketmq-broker-0, queueId=2]" underlayError="the broker [rocketmq-broker-0] does not exist"
time="2022-08-18T15:37:55+08:00" level=warning msg="do defaultConsumer, add a new mq failed" MessageQueue="MessageQueue [topic=slave-test, brokerName=rocketmq-broker-0, queueId=2]" consumerGroup=slave-test
time="2022-08-18T15:37:55+08:00" level=warning msg="do balance in group failed, the topic does not exist" consumerGroup=slave-test topic="%RETRY%slave-test"
time="2022-08-18T15:37:55+08:00" level=warning msg="delete mq from offset table" MessageQueue="MessageQueue [topic=slave-test, brokerName=rocketmq-broker-0, queueId=0]" consumerGroup=slave-test
time="2022-08-18T15:37:55+08:00" level=error msg="fecth offset of mq from broker error" MessageQueue="MessageQueue [topic=slave-test, brokerName=rocketmq-broker-0, queueId=0]" consumerGroup=slave-test underlayError="broker: rocke
tmq-broker-0 address not found"
time="2022-08-18T15:37:55+08:00" level=warning msg="query max offset error" MessageQueue="MessageQueue [topic=slave-test, brokerName=rocketmq-broker-0, queueId=0]" underlayError="the broker [rocketmq-broker-0] does not exist"
time="2022-08-18T15:37:55+08:00" level=warning msg="do defaultConsumer, add a new mq failed" MessageQueue="MessageQueue [topic=slave-test, brokerName=rocketmq-broker-0, queueId=0]" consumerGroup=slave-test
time="2022-08-18T15:37:55+08:00" level=warning msg="delete mq from offset table" MessageQueue="MessageQueue [topic=slave-test, brokerName=rocketmq-broker-0, queueId=1]" consumerGroup=slave-test
time="2022-08-18T15:37:55+08:00" level=error msg="fecth offset of mq from broker error" MessageQueue="MessageQueue [topic=slave-test, brokerName=rocketmq-broker-0, queueId=1]" consumerGroup=slave-test underlayError="broker: rocke
tmq-broker-0 address not found"
time="2022-08-18T15:37:55+08:00" level=warning msg="query max offset error" MessageQueue="MessageQueue [topic=slave-test, brokerName=rocketmq-broker-0, queueId=1]" underlayError="the broker [rocketmq-broker-0] does not exist"
time="2022-08-18T15:37:55+08:00" level=warning msg="do defaultConsumer, add a new mq failed" MessageQueue="MessageQueue [topic=slave-test, brokerName=rocketmq-broker-0, queueId=1]" consumerGroup=slave-test
time="2022-08-18T15:37:55+08:00" level=warning msg="delete mq from offset table" MessageQueue="MessageQueue [topic=slave-test, brokerName=rocketmq-broker-0, queueId=2]" consumerGroup=slave-test
time="2022-08-18T15:37:55+08:00" level=error msg="fecth offset of mq from broker error" MessageQueue="MessageQueue [topic=slave-test, brokerName=rocketmq-broker-0, queueId=2]" consumerGroup=slave-test underlayError="broker: rocke
tmq-broker-0 address not found"
time="2022-08-18T15:37:55+08:00" level=warning msg="query max offset error" MessageQueue="MessageQueue [topic=slave-test, brokerName=rocketmq-broker-0, queueId=2]" underlayError="the broker [rocketmq-broker-0] does not exist"
time="2022-08-18T15:37:55+08:00" level=warning msg="do defaultConsumer, add a new mq failed" MessageQueue="MessageQueue [topic=slave-test, brokerName=rocketmq-broker-0, queueId=2]" consumerGroup=slave-test
time="2022-08-18T15:37:55+08:00" level=warning msg="delete mq from offset table" MessageQueue="MessageQueue [topic=slave-test, brokerName=rocketmq-broker-0, queueId=3]" consumerGroup=slave-test
time="2022-08-18T15:37:55+08:00" level=error msg="fecth offset of mq from broker error" MessageQueue="MessageQueue [topic=slave-test, brokerName=rocketmq-broker-0, queueId=3]" consumerGroup=slave-test underlayError="broker: rocke
tmq-broker-0 address not found"
time="2022-08-18T15:37:55+08:00" level=warning msg="query max offset error" MessageQueue="MessageQueue [topic=slave-test, brokerName=rocketmq-broker-0, queueId=3]" underlayError="the broker [rocketmq-broker-0] does not exist"
time="2022-08-18T15:37:55+08:00" level=warning msg="do defaultConsumer, add a new mq failed" MessageQueue="MessageQueue [topic=slave-test, brokerName=rocketmq-broker-0, queueId=3]" consumerGroup=slave-test
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] RongtongJin commented on issue #4840: In HA mode, the master is down and cannot be consumed from the slave
Posted by GitBox <gi...@apache.org>.
RongtongJin commented on issue #4840:
URL: https://github.com/apache/rocketmq/issues/4840#issuecomment-1219152962
Hi, @MrLYG two conditions need to be met for consumption from slave broker: 1. The configuration slavReadEnable must be true 2. It cannot be the consumption of order messages.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] MrLYG commented on issue #4840: In HA mode, the master is down and cannot be consumed from the slave
Posted by GitBox <gi...@apache.org>.
MrLYG commented on issue #4840:
URL: https://github.com/apache/rocketmq/issues/4840#issuecomment-1223804237
> > That's this means the go client has some bug or problem?
> > If I need to use the go client to connect the rocektmq service, what should I do to avoid that problem? Could you give me some suggestions?
>
> IIRC, the latest version of rocketmq-client-go has fixed this issue. @guyinyou @ShannonDing Could you help to take a look？
I used the latest version of rocketmq-client-go to consume messages from slave but still fail.
There is my go.mod file in which I require the latest version of rocketmq-client-go 2.1.1
```
module go-rocket
go 1.17
require github.com/apache/rocketmq-client-go/v2 v2.1.1
require (
github.com/emirpasic/gods v1.12.0 // indirect
github.com/golang/mock v1.3.1 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/konsorten/go-windows-terminal-sequences v1.0.1 // indirect
github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421 // indirect
github.com/modern-go/reflect2 v1.0.2 // indirect
github.com/patrickmn/go-cache v2.1.0+incompatible // indirect
github.com/pkg/errors v0.8.1 // indirect
github.com/satori/go.uuid v1.2.0 // indirect
github.com/sirupsen/logrus v1.4.0 // indirect
github.com/stathat/consistent v1.0.0 // indirect
github.com/tidwall/gjson v1.13.0 // indirect
github.com/tidwall/match v1.1.1 // indirect
github.com/tidwall/pretty v1.2.0 // indirect
go.uber.org/atomic v1.5.1 // indirect
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2 // indirect
golang.org/x/lint v0.0.0-20190930215403-16217165b5de // indirect
golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f // indirect
golang.org/x/tools v0.0.0-20191029041327-9cc4af7d6b2c // indirect
gopkg.in/natefinch/lumberjack.v2 v2.0.0 // indirect
)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] RongtongJin commented on issue #4840: In HA mode, the master is down and cannot be consumed from the slave
Posted by GitBox <gi...@apache.org>.
RongtongJin commented on issue #4840:
URL: https://github.com/apache/rocketmq/issues/4840#issuecomment-1219228193
> > Hi, @MrLYG two conditions need to be met for consumption from slave broker: 1. The configuration slavReadEnable must be true 2. It cannot be the consumption of order messages.
>
> @RongtongJin Thanks a ton for your fastly reply. I had met the two conditions that you gave. But the consumer still not consume from slave when master done.
>
> 1. The configuration slavReadEnable is true
> 
> 2. I didn't open the consumption of order messages.
> 
@MrLYG Could you try whether the Java client has the same issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] RongtongJin commented on issue #4840: In HA mode, the master is down and cannot be consumed from the slave
Posted by GitBox <gi...@apache.org>.
RongtongJin commented on issue #4840:
URL: https://github.com/apache/rocketmq/issues/4840#issuecomment-1223462696
> That's this means the go client has some bug or problem?
> If I need to use the go client to connect the rocektmq service, what should I do to avoid that problem? Could you give me some suggestions?
IIRC, the latest version of rocketmq-client-go has fixed this issue. @guyinyou @ShannonDing Could you help to take a look？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] github-actions[bot] commented on issue #4840: In HA mode, the master is down and cannot be consumed from the slave
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #4840:
URL: https://github.com/apache/rocketmq/issues/4840#issuecomment-1694527595
This issue was closed because it has been inactive for 3 days since being marked as stale.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] github-actions[bot] closed issue #4840: In HA mode, the master is down and cannot be consumed from the slave
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #4840: In HA mode, the master is down and cannot be consumed from the slave
URL: https://github.com/apache/rocketmq/issues/4840
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] MrLYG commented on issue #4840: In HA mode, the master is down and cannot be consumed from the slave
Posted by GitBox <gi...@apache.org>.
MrLYG commented on issue #4840:
URL: https://github.com/apache/rocketmq/issues/4840#issuecomment-1220223306
> > > Hi, @MrLYG two conditions need to be met for consumption from slave broker: 1. The configuration slavReadEnable must be true 2. It cannot be the consumption of order messages.
> >
> >
> > @RongtongJin Thanks a ton for your fastly reply. I had met the two conditions that you gave. But the consumer still not consume from slave when master done.
> >
> > 1. The configuration slavReadEnable is true
> > 
> > 2. I didn't open the consumption of order messages.
> > 
>
> @MrLYG Could you try whether the Java client has the same issue?
The Java client can consume messages when the master has done.
That's this means the go client has some bug or problem?
If I need to use the go client to connect the rocektmq service, what should I do to avoid that problem? Could you give me some suggestions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] MrLYG commented on issue #4840: In HA mode, the master is down and cannot be consumed from the slave
Posted by GitBox <gi...@apache.org>.
MrLYG commented on issue #4840:
URL: https://github.com/apache/rocketmq/issues/4840#issuecomment-1219177942
> Hi, @MrLYG two conditions need to be met for consumption from slave broker: 1. The configuration slavReadEnable must be true 2. It cannot be the consumption of order messages.
Thanks a ton for your fastly reply.
1. The configuration slavReadEnable is true
2. I didn't open the consumption of order messages.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] github-actions[bot] commented on issue #4840: In HA mode, the master is down and cannot be consumed from the slave
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #4840:
URL: https://github.com/apache/rocketmq/issues/4840#issuecomment-1690797433
This issue is stale because it has been open for 365 days with no activity. It will be closed in 3 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org