You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@rocketmq.apache.org by "zhongxiaoyi738 (via GitHub)" <gi...@apache.org> on 2025/02/28 07:33:20 UTC

[D] rocketMQ4.9.2 不同broker之间的consumer offset没有同步，enableDLegerCommitLog=true [rocketmq]

GitHub user zhongxiaoyi738 created a discussion: rocketMQ4.9.2 不同broker之间的consumer offset没有同步，enableDLegerCommitLog=true

**[rocketmq@broker-01-68d4859b4d-b7gq7 bin]$ more /opt/conf/broker-01.conf** 
brokerClusterName = DefaultCluster
brokerName=broker-01
listenPort=31863
brokerIP1=10.7.30.237
autoCreateTopicEnable=true
autoCreateSubscriptionGroup=true
maxMessageSize=65535
flushDiskType=ASYNC_FLUSH
deleteWhen=04
fileReservedTime=48
mapedFileSizeCommitLog=1073741824
mapedFileSizeConsumeQueue=30000
diskMaxUsedSpaceRatio=88
storePathRootDir=/home/rocketmq/store/node00
storePathCommitLog=/home/rocketmq/store/node00/commitlog
enableDLegerCommitLog=true
dLegerGroup=RaftNode00
dLegerPeers=n1-broker-01.rocketmq-flow.svc.cluster.local:40911;n2-broker-02.rocketmq-flow.svc.cluster.local:40911;n3-broker-03.rocketmq-flow.svc.cluster.local:40911
## must be unique
dLegerSelfId=n1
sendMessageThreadPoolNums=16

**[rocketmq@broker-02-84cc5b59fb-d2n2r bin]$ more /opt/conf/broker-02.conf** 
brokerClusterName = DefaultCluster
brokerName=broker-02
listenPort=31866
brokerIP1=10.7.30.237
autoCreateTopicEnable=true
autoCreateSubscriptionGroup=true
maxMessageSize=65535
flushDiskType=ASYNC_FLUSH
deleteWhen=04
fileReservedTime=48
mapedFileSizeCommitLog=1073741824
mapedFileSizeConsumeQueue=30000
diskMaxUsedSpaceRatio=88
storePathRootDir=/home/rocketmq/store/node00
storePathCommitLog=/home/rocketmq/store/node00/commitlog
enableDLegerCommitLog=true
dLegerGroup=RaftNode00
dLegerPeers=n1-broker-01.rocketmq-flow.svc.cluster.local:40911;n2-broker-02.rocketmq-flow.svc.cluster.local:40911;n3-broker-03.rocketmq-flow.svc.cluster.local:40911
## must be unique
dLegerSelfId=n2
sendMessageThreadPoolNums=16

**[rocketmq@broker-03-584bb8879d-5b9tq bin]$ more /opt/conf/broker-03.conf** 
brokerClusterName = DefaultCluster
brokerName=broker-03
listenPort=31869
brokerIP1=10.7.30.237
autoCreateTopicEnable=true
autoCreateSubscriptionGroup=true
maxMessageSize=65535
flushDiskType=ASYNC_FLUSH
deleteWhen=04
fileReservedTime=48
mapedFileSizeCommitLog=1073741824
mapedFileSizeConsumeQueue=30000
diskMaxUsedSpaceRatio=88
storePathRootDir=/home/rocketmq/store/node00
storePathCommitLog=/home/rocketmq/store/node00/commitlog
enableDLegerCommitLog=true
dLegerGroup=RaftNode00
dLegerPeers=n1-broker-01.rocketmq-flow.svc.cluster.local:40911;n2-broker-02.rocketmq-flow.svc.cluster.local:40911;n3-broker-03.rocketmq-flow.svc.cluster.local:40911
## must be unique
dLegerSelfId=n3
sendMessageThreadPoolNums=16


**broker-02**
#Topic                            #Group                            #Broker Name                      #QID  #Broker Offset        #Consumer Offset      #Diff                 #LastTime
ANALYSIS_TEMPLATE_TO_CARD_UPD     ANALYSIS_TEMPLATE_TO_CARD_UPD     broker-02                         0     133                   133                   0                     2025-01-19 14:43:05
ANALYSIS_TEMPLATE_TO_CARD_UPD     ANALYSIS_TEMPLATE_TO_CARD_UPD     broker-02                         1     121                   121                   0                     2025-01-20 15:13:56
ANALYSIS_TEMPLATE_TO_CARD_UPD     ANALYSIS_TEMPLATE_TO_CARD_UPD     broker-02                         2     135                   134                   1                     2025-01-19 14:43:05
ANALYSIS_TEMPLATE_TO_CARD_UPD     ANALYSIS_TEMPLATE_TO_CARD_UPD     broker-02                         3     121                   121                   0                     2025-01-19 14:43:35


broker-03
#Topic                            #Group                            #Broker Name                      #QID  #Broker Offset        #Consumer Offset      #Diff                 #LastTime
ANALYSIS_TEMPLATE_TO_CARD_UPD     ANALYSIS_TEMPLATE_TO_CARD_UPD     broker-03                         0     133                   133                   0                     2025-01-19 14:43:05
ANALYSIS_TEMPLATE_TO_CARD_UPD     ANALYSIS_TEMPLATE_TO_CARD_UPD     broker-03                         1     121                   121                   0                     2025-01-20 15:13:56
ANALYSIS_TEMPLATE_TO_CARD_UPD     ANALYSIS_TEMPLATE_TO_CARD_UPD     broker-03                         2     135                   135                   0                     2025-02-27 16:39:54
ANALYSIS_TEMPLATE_TO_CARD_UPD     ANALYSIS_TEMPLATE_TO_CARD_UPD     broker-03                         3     121                   121                   0                     2025-01-19 14:43:35

GitHub link: https://github.com/apache/rocketmq/discussions/9216

----
This is an automatically sent email for dev@rocketmq.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@rocketmq.apache.org

Re: [D] rocketMQ4.9.2 不同broker之间的consumer offset没有同步，enableDLegerCommitLog=true [rocketmq]

Posted by "zhongxiaoyi738 (via GitHub)" <gi...@apache.org>.
GitHub user zhongxiaoyi738 added a comment to the discussion: rocketMQ4.9.2 不同broker之间的consumer offset没有同步，enableDLegerCommitLog=true

nameserver && broker 都没有报错日志哦

GitHub link: https://github.com/apache/rocketmq/discussions/9216#discussioncomment-12446251

----
This is an automatically sent email for dev@rocketmq.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@rocketmq.apache.org

Re: [D] rocketMQ4.9.2 不同broker之间的consumer offset没有同步，enableDLegerCommitLog=true [rocketmq]

Posted by "francisoliverlee (via GitHub)" <gi...@apache.org>.
GitHub user francisoliverlee added a comment to the discussion: rocketMQ4.9.2 不同broker之间的consumer offset没有同步，enableDLegerCommitLog=true

DLeger模式只涉及commitlog是raft的， 元数据应该还是一样同步。不过貌似有个bug，dleger模式不能为null，不过我没有验证， 你可以打日志check下
https://github.com/apache/rocketmq/blob/c840b208337b26c7529602b81d398c0a87a23888/broker/src/main/java/org/apache/rocketmq/broker/BrokerController.java#L1139

GitHub link: https://github.com/apache/rocketmq/discussions/9216#discussioncomment-12469162

----
This is an automatically sent email for dev@rocketmq.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@rocketmq.apache.org

Re: [D] rocketMQ4.9.2 不同broker之间的consumer offset没有同步，enableDLegerCommitLog=true [rocketmq]

Posted by "zhongxiaoyi738 (via GitHub)" <gi...@apache.org>.
GitHub user zhongxiaoyi738 added a comment to the discussion: rocketMQ4.9.2 不同broker之间的consumer offset没有同步，enableDLegerCommitLog=true

补充问题
rocketMQ4.9.2，开启enableDLegerCommitLog=true
topic、consumerOffset等元数据，是如何同步的？

GitHub link: https://github.com/apache/rocketmq/discussions/9216#discussioncomment-12458948

----
This is an automatically sent email for dev@rocketmq.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@rocketmq.apache.org

Re: [D] rocketMQ4.9.2 不同broker之间的consumer offset没有同步，enableDLegerCommitLog=true [rocketmq]

Posted by "zhongxiaoyi738 (via GitHub)" <gi...@apache.org>.
GitHub user zhongxiaoyi738 edited a comment on the discussion: rocketMQ4.9.2 不同broker之间的consumer offset没有同步，enableDLegerCommitLog=true

请问你提到的这个bug有修复么？

我加了调试日志
2025-03-11 15:45:33 INFO BrokerControllerScheduledThread1 - xiaoyi----syncTopicConfig masterAddrBak=null, brokerController.getBrokerAddr()=10.2.30.171:31879

GitHub link: https://github.com/apache/rocketmq/discussions/9216#discussioncomment-12471129

----
This is an automatically sent email for dev@rocketmq.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@rocketmq.apache.org

Re: [D] rocketMQ4.9.2 不同broker之间的consumer offset没有同步，enableDLegerCommitLog=true [rocketmq]

Posted by "zhongxiaoyi738 (via GitHub)" <gi...@apache.org>.
GitHub user zhongxiaoyi738 added a comment to the discussion: rocketMQ4.9.2 不同broker之间的consumer offset没有同步，enableDLegerCommitLog=true

补充，从节点的日志摘录（刚刚启动时有报错，但是后续恢复正常了）

[rocketmq@broker-02-7dcffbb679-jmxlb rocketmqlogs]$ more broker_default.log 
2025-03-11 16:10:46 INFO main - instrumented a special java.util.Set into: sun.nio.ch.EPollSelectorImpl@458c1321

2025-03-11 16:11:46 INFO voteInvokeExecutor_3 - [n2][GetVoteResponse] {"code":200,"term":-1,"voteResult":"UNKNOWN"}
2025-03-11 16:11:47 INFO QuorumAckChecker-n2 - [n2][CANDIDATE] term=24 ledgerBegin=1374454 ledgerEnd=1746357 committed=1746357 watermarks={}
2025-03-11 16:11:49 INFO StateMaintainer - [n2] [PARSE_VOTE_RESULT] cost=2600 term=24 memberNum=3 allNum=2 acceptedNum=0 notReadyTermNum=0 biggerLedgerNum=0 alreadyHasLeader=false maxTerm=24 result=WAIT_TO_REVOTE
2025-03-11 16:11:49 INFO NettyServerPublicExecutor_3 - [n2][ChangeRoleToFollower] from term: 24 leaderId: n1 and currTerm: 24
2025-03-11 16:11:49 ERROR voteInvokeExecutor_4 - Send vote request failed, info[group=RaftNode00,term=24,code=200,local=null,remote=n3,leader=n2]
org.apache.rocketmq.remoting.exception.RemotingConnectException: connect to broker-03.rocketmq-flow.svc.cluster.local:40911 failed
at org.apache.rocketmq.remoting.netty.NettyRemotingClient.invokeAsync(NettyRemotingClient.java:535) ~[rocketmq-remoting-4.9.2.jar:4.9.2]
        at io.openmessaging.storage.dledger.DLedgerRpcNettyService.lambda$vote$3(DLedgerRpcNettyService.java:164) ~[dledger-0.2.2.jar:na]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_322]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_322]
        at java.lang.Thread.run(Thread.java:750) ~[na:1.8.0_322]
2025-03-11 16:11:49 INFO voteInvokeExecutor_4 - [n2][GetVoteResponse] {"code":200,"term":-1,"voteResult":"UNKNOWN"}
2025-03-11 16:11:50 INFO QuorumAckChecker-n2 - [n2][FOLLOWER] term=24 ledgerBegin=1374454 ledgerEnd=1746357 committed=1746357 watermarks={}
2025-03-11 16:11:53 INFO QuorumAckChecker-n2 - [n2][FOLLOWER] term=24 ledgerBegin=1374454 ledgerEnd=1746357 committed=1746357 watermarks={}
后面一直打印的日志，跟最后一行一致

GitHub link: https://github.com/apache/rocketmq/discussions/9216#discussioncomment-12458871

----
This is an automatically sent email for dev@rocketmq.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@rocketmq.apache.org

Re: [D] rocketMQ4.9.2 不同broker之间的consumer offset没有同步，enableDLegerCommitLog=true [rocketmq]

Posted by "zhongxiaoyi738 (via GitHub)" <gi...@apache.org>.
GitHub user zhongxiaoyi738 added a comment to the discussion: rocketMQ4.9.2 不同broker之间的consumer offset没有同步，enableDLegerCommitLog=true

brokerClusterName = DefaultCluster
**brokerName=broker-01**
listenPort=31863
brokerIP1=10.7.30.237
autoCreateTopicEnable=true
autoCreateSubscriptionGroup=true
maxMessageSize=65535
flushDiskType=ASYNC_FLUSH
deleteWhen=04
fileReservedTime=48
mapedFileSizeCommitLog=1073741824
mapedFileSizeConsumeQueue=30000
diskMaxUsedSpaceRatio=88
storePathRootDir=/home/rocketmq/store/node00
storePathCommitLog=/home/rocketmq/store/node00/commitlog
enableDLegerCommitLog=true
**dLegerGroup=RaftNode00**
dLegerPeers=**n1-broker-01**.rocketmq-flow.svc.cluster.local:40911;n2-broker-02.rocketmq-flow.svc.cluster.local:40911;n3-broker-03.rocketmq-flow.svc.cluster.local:40911
**dLegerSelfId=n1**
sendMessageThreadPoolNums=16

1、brokerName、dLegerGroup不一致
2、dLegerPeers 与 dLegerSelfId 不一致
请问这2个不一致会有影响么？
感觉BrokerController.handleSlaveSynchronize这个定时任务没有启动哦，jstack打印出来BrokerController只有一个线程

GitHub link: https://github.com/apache/rocketmq/discussions/9216#discussioncomment-12446867

----
This is an automatically sent email for dev@rocketmq.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@rocketmq.apache.org

Re: [D] rocketMQ4.9.2 不同broker之间的consumer offset没有同步，enableDLegerCommitLog=true [rocketmq]

Posted by "francisoliverlee (via GitHub)" <gi...@apache.org>.
GitHub user francisoliverlee added a comment to the discussion: rocketMQ4.9.2 不同broker之间的consumer offset没有同步，enableDLegerCommitLog=true

1、brokerName、dLegerGroup不一致
不影响

2、dLegerPeers 与 dLegerSelfId 不一致
dLegerSelfId是dLegerPeers的一部分。 具体看[这里](https://rocketmq.apache.org/zh/docs/bestPractice/02dledger#1-%E6%96%B0%E9%9B%86%E7%BE%A4%E9%83%A8%E7%BD%B2)

check [peer state change log in codes](https://github.com/apache/rocketmq/blob/c840b208337b26c7529602b81d398c0a87a23888/broker/src/main/java/org/apache/rocketmq/broker/BrokerController.java#L1160-L1226)

GitHub link: https://github.com/apache/rocketmq/discussions/9216#discussioncomment-12456891

----
This is an automatically sent email for dev@rocketmq.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@rocketmq.apache.org

Re: [D] rocketMQ4.9.2 不同broker之间的consumer offset没有同步，enableDLegerCommitLog=true [rocketmq]

Posted by "zhongxiaoyi738 (via GitHub)" <gi...@apache.org>.
GitHub user zhongxiaoyi738 added a comment to the discussion: rocketMQ4.9.2 不同broker之间的consumer offset没有同步，enableDLegerCommitLog=true

dLegerPeers=n1-broker-01.rocketmq-flow.svc.cluster.local:40911;n2-broker-02.rocketmq-flow.svc.cluster.local:40911;n3-broker-03.rocketmq-flow.svc.cluster.local:40911
dLegerSelfId=n1

我们这样配置，对么？
如果用域名，应该如何配置

GitHub link: https://github.com/apache/rocketmq/discussions/9216#discussioncomment-12458693

----
This is an automatically sent email for dev@rocketmq.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@rocketmq.apache.org

Re: [D] rocketMQ4.9.2 不同broker之间的consumer offset没有同步，enableDLegerCommitLog=true [rocketmq]

Posted by "francisoliverlee (via GitHub)" <gi...@apache.org>.
GitHub user francisoliverlee added a comment to the discussion: rocketMQ4.9.2 不同broker之间的consumer offset没有同步，enableDLegerCommitLog=true

any error logs? 
[DLeger Model](DLeger)  not suggest, try [Controller Mode](https://rocketmq.apache.org/zh/docs/deploymentOperations/03autofailover/) 


GitHub link: https://github.com/apache/rocketmq/discussions/9216#discussioncomment-12370050

----
This is an automatically sent email for dev@rocketmq.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@rocketmq.apache.org

Re: [D] rocketMQ4.9.2 不同broker之间的consumer offset没有同步，enableDLegerCommitLog=true [rocketmq]

Posted by "zhongxiaoyi738 (via GitHub)" <gi...@apache.org>.
GitHub user zhongxiaoyi738 added a comment to the discussion: rocketMQ4.9.2 不同broker之间的consumer offset没有同步，enableDLegerCommitLog=true

我加了调试日志
2025-03-11 15:45:33 INFO BrokerControllerScheduledThread1 - xiaoyi----syncTopicConfig masterAddrBak=null, brokerController.getBrokerAddr()=10.2.30.171:31879

masterAddrBak为空，请问是哪里配置不对么？

GitHub link: https://github.com/apache/rocketmq/discussions/9216#discussioncomment-12471129

----
This is an automatically sent email for dev@rocketmq.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@rocketmq.apache.org

Re: [D] rocketMQ4.9.2 不同broker之间的consumer offset没有同步，enableDLegerCommitLog=true [rocketmq]

Posted by "zhongxiaoyi738 (via GitHub)" <gi...@apache.org>.
GitHub user zhongxiaoyi738 added a comment to the discussion: rocketMQ4.9.2 不同broker之间的consumer offset没有同步，enableDLegerCommitLog=true

hi，我改了一下rocketMQ的源码（加了很多调试日志），新增加的日志如下：（masterAddrBak=null）

2025-03-11 15:45:30 INFO DLegerRoleChangeHandler_1 - xiaoyi----handleSlaveSynchronize role=SLAVE
2025-03-11 15:45:30 INFO DLegerRoleChangeHandler_1 - xiaoyi----handleSlaveSynchronize ready slaveSyncFuture
2025-03-11 15:45:33 INFO BrokerControllerScheduledThread1 - xiaoyi----handleSlaveSynchronize run syncAll
2025-03-11 15:45:33 INFO BrokerControllerScheduledThread1 - xiaoyi----syncTopicConfig     **masterAddrBak=null**, brokerController.getBrokerAddr()=10.2.30.171:31879
2025-03-11 15:45:33 INFO BrokerControllerScheduledThread1 - xiaoyi----syncConsumerOffset     masterAddrBak=null, brokerController.getBrokerAddr()=10.2.30.171:31879
2025-03-11 15:45:33 INFO BrokerControllerScheduledThread1 - xiaoyi----syncDelayOffset     masterAddrBak=null, brokerController.getBrokerAddr()=10.2.30.171:31879
2025-03-11 15:45:33 INFO BrokerControllerScheduledThread1 - xiaoyi----syncSubscriptionGroupConfig     masterAddrBak=null, brokerController.getBrokerAddr()=10.2.30.171:31879

GitHub link: https://github.com/apache/rocketmq/discussions/9216#discussioncomment-12458652

----
This is an automatically sent email for dev@rocketmq.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@rocketmq.apache.org

Re: [D] rocketMQ4.9.2 不同broker之间的consumer offset没有同步，enableDLegerCommitLog=true [rocketmq]

Posted by "francisoliverlee (via GitHub)" <gi...@apache.org>.
GitHub user francisoliverlee added a comment to the discussion: rocketMQ4.9.2 不同broker之间的consumer offset没有同步，enableDLegerCommitLog=true

满足这个逻辑即可
![Clipboard_Screenshot_1741745972](https://github.com/user-attachments/assets/706b59dd-b92c-4bad-ad5c-c4281243e37a)


GitHub link: https://github.com/apache/rocketmq/discussions/9216#discussioncomment-12468955

----
This is an automatically sent email for dev@rocketmq.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@rocketmq.apache.org