You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@rocketmq.apache.org by GitBox <gi...@apache.org> on 2021/01/08 07:55:49 UTC
[GitHub] [rocketmq] liuyk0109 opened a new issue #2574: The 4.8.0 version of the dledger mode does not work in k8s
liuyk0109 opened a new issue #2574:
URL: https://github.com/apache/rocketmq/issues/2574
I build the namesrv and broker container myself and run them on k8s, and use the dledger mode. They can start and look like normal.
But when i send a message with mqadmin, some exceptions were thrown.
I run the cluster on k8s namespace rocketmq, there is my broker config file:
n0:
```
brokerClusterName = RaftCluster
brokerName=RaftNode00
brokerIP1=rocketmq-broker-0.rocketmq-broker-service.rocketmq
brokerIP2=rocketmq-broker-0.rocketmq-broker-service.rocketmq
listenPort=10911
namesrvAddr=rocketmq-namesrv-0.rocketmq-namesrv-service:9876;rocketmq-namesrv-1.rocketmq-namesrv-service:9876;rocketmq-namesrv-2.rocketmq-namesrv-service:9876
storePathRootDir=/mountdir/rmqstore/n0
storePathCommitLog=/mountdir/rmqstore/n0/commitlog
enableDLegerCommitLog=true
dLegerGroup=RaftNode00
dLegerPeers=n0-rocketmq-broker-0.rocketmq-broker-service:20911;n1-rocketmq-broker-1.rocketmq-broker-service:20911;n2-rocketmq-broker-2.rocketmq-broker-service:20911
## must be unique
dLegerSelfId=n0
sendMessageThreadPoolNums=16
autoCreateTopicEnable=false
```
n1:
```
brokerClusterName = RaftCluster
brokerName=RaftNode00
brokerIP1=rocketmq-broker-1.rocketmq-broker-service.rocketmq
brokerIP2=rocketmq-broker-1.rocketmq-broker-service.rocketmq
listenPort=10911
namesrvAddr=rocketmq-namesrv-0.rocketmq-namesrv-service:9876;rocketmq-namesrv-1.rocketmq-namesrv-service:9876;rocketmq-namesrv-2.rocketmq-namesrv-service:9876
storePathRootDir=/mountdir/rmqstore/n1
storePathCommitLog=/mountdir/rmqstore/n1/commitlog
enableDLegerCommitLog=true
dLegerGroup=RaftNode00
dLegerPeers=n0-rocketmq-broker-0.rocketmq-broker-service:20911;n1-rocketmq-broker-1.rocketmq-broker-service:20911;n2-rocketmq-broker-2.rocketmq-broker-service:20911
## must be unique
dLegerSelfId=n1
sendMessageThreadPoolNums=16
autoCreateTopicEnable=false
```
n2:
```
brokerClusterName = RaftCluster
brokerName=RaftNode00
brokerIP1=rocketmq-broker-2.rocketmq-broker-service.rocketmq
brokerIP2=rocketmq-broker-2.rocketmq-broker-service.rocketmq
listenPort=10911
namesrvAddr=rocketmq-namesrv-0.rocketmq-namesrv-service:9876;rocketmq-namesrv-1.rocketmq-namesrv-service:9876;rocketmq-namesrv-2.rocketmq-namesrv-service:9876
storePathRootDir=/mountdir/rmqstore/n2
storePathCommitLog=/mountdir/rmqstore/n2/commitlog
enableDLegerCommitLog=true
dLegerGroup=RaftNode00
dLegerPeers=n0-rocketmq-broker-0.rocketmq-broker-service:20911;n1-rocketmq-broker-1.rocketmq-broker-service:20911;n2-rocketmq-broker-2.rocketmq-broker-service:20911
## must be unique
dLegerSelfId=n2
sendMessageThreadPoolNums=16
autoCreateTopicEnable=false
```
The `/mountdir` is nfs pvc path.
Some logs in the n0 broker that I think are useful:
```
2021-01-08 14:30:47 INFO brokerOutApi_thread_1 - register broker[1]to name server rocketmq-namesrv-1.rocketmq-namesrv-service:9876 OK
2021-01-08 14:30:47 INFO brokerOutApi_thread_4 - register broker[1]to name server rocketmq-namesrv-0.rocketmq-namesrv-service:9876 OK
2021-01-08 14:30:47 INFO brokerOutApi_thread_2 - register broker[1]to name server rocketmq-namesrv-2.rocketmq-namesrv-service:9876 OK
2021-01-08 14:30:47 ERROR DiskCheckScheduledThread1 - Error when measuring disk space usage, file doesn't exist on this path: /mountdir/rmqstore/n0/commitlog
2021-01-08 14:30:52 INFO BrokerControllerScheduledThread1 - Update slave topic config from master, rocketmq-broker-1.rocketmq-broker-service.rocketmq:10911
2021-01-08 14:30:53 INFO BrokerControllerScheduledThread1 - Update slave consumer offset from master, rocketmq-broker-1.rocketmq-broker-service.rocketmq:10911
2021-01-08 14:30:53 INFO BrokerControllerScheduledThread1 - Update slave delay offset from master, rocketmq-broker-1.rocketmq-broker-service.rocketmq:10911
2021-01-08 14:30:53 INFO BrokerControllerScheduledThread1 - Update slave Subscription Group from master, rocketmq-broker-1.rocketmq-broker-service.rocketmq:10911
2021-01-08 14:30:57 ERROR DiskCheckScheduledThread1 - Error when measuring disk space usage, file doesn't exist on this path: /mountdir/rmqstore/n0/commitlog
2021-01-08 14:31:04 INFO BrokerControllerScheduledThread1 - Update slave consumer offset from master, rocketmq-broker-1.rocketmq-broker-service.rocketmq:10911
2021-01-08 14:31:04 INFO BrokerControllerScheduledThread1 - Update slave delay offset from master, rocketmq-broker-1.rocketmq-broker-service.rocketmq:10911
2021-01-08 14:31:06 ERROR StoreScheduledThread1 - Error when measuring disk space usage, file doesn't exist on this path: /mountdir/rmqstore/n0/dledger-n0/data
```
Some files can't find:
```
[root@hadoop-dn02 rocketmq]# ll rmqstore/n0/
-rw-r--r-- 1 root root 0 1月 8 14:30 abort
-rw-r--r-- 1 root root 4096 1月 8 15:40 checkpoint
drwxr-xr-x 2 root root 280 1月 8 15:40 config
drwxr-xr-x 2 root root 82 1月 8 15:40 dledger-n0
-rw-r--r-- 1 root root 4 1月 8 14:30 lock
[root@hadoop-dn02 rocketmq]# ll rmqstore/n0/dledger-n0/
-rw-r--r-- 1 root root 30 1月 8 15:41 checkpoint
-rw-r--r-- 1 root root 30 1月 8 15:41 checkpoint.bak
-rw-r--r-- 1 root root 25 1月 8 14:30 currterm
-rw-r--r-- 1 root root 23 1月 8 14:30 currterm.bak
```
Entry the broker n0 container and run some commands by mqadmin:
clusterList:
```
[root@rocketmq-broker-0 bin]# sh mqadmin clusterList -n rocketmq-namesrv-0.rocketmq-namesrv-service.rocketmq:9876
RocketMQLog:WARN No appenders could be found for logger (io.netty.util.internal.PlatformDependent0).
RocketMQLog:WARN Please initialize the logger system properly.
#Cluster Name #Broker Name #BID #Addr #Version #InTPS(LOAD) #OutTPS(LOAD) #PCWait(ms) #Hour #SPACE
RaftCluster RaftNode00 0 rocketmq-broker-1.rocketmq-broker-service.rocketmq:10911 V4_8_0 0.00(0,0ms) 0.00(0,0ms) 0 447246.58 -1.0000
RaftCluster RaftNode00 1 rocketmq-broker-0.rocketmq-broker-service.rocketmq:10911 V4_8_0 0.00(0,0ms) 0.00(0,0ms) 0 447246.58 -1.0000
RaftCluster RaftNode00 3 rocketmq-broker-2.rocketmq-broker-service.rocketmq:10911 V4_8_0 0.00(0,0ms) 0.00(0,0ms) 0 447246.58 -1.0000
```
updateTopic:
```
[root@rocketmq-broker-0 bin]# sh mqadmin updateTopic -n rocketmq-namesrv-0.rocketmq-namesrv-service.rocketmq:9876 -c RaftCluster -t lyktest
RocketMQLog:WARN No appenders could be found for logger (io.netty.util.internal.PlatformDependent0).
RocketMQLog:WARN Please initialize the logger system properly.
create topic to rocketmq-broker-1.rocketmq-broker-service.rocketmq:10911 success.
TopicConfig [topicName=lyktest, readQueueNums=8, writeQueueNums=8, perm=RW-, topicFilterType=SINGLE_TAG, topicSysFlag=0, order=false]
```
topicList:
```
sh mqadmin topicList -n rocketmq-namesrv-0.rocketmq-namesrv-service.rocketmq:9876
RocketMQLog:WARN No appenders could be found for logger (io.netty.util.internal.PlatformDependent0).
RocketMQLog:WARN Please initialize the logger system properly.
RaftCluster_REPLY_TOPIC
SCHEDULE_TOPIC_XXXX
RMQ_SYS_TRANS_HALF_TOPIC
lyktest
RaftCluster
BenchmarkTest
OFFSET_MOVED_EVENT
RaftNode00
SELF_TEST_TOPIC
```
Everything looks normal, but when i send a message:
```
sh mqadmin sendMessage -n rocketmq-namesrv-0.rocketmq-namesrv-service.rocketmq:9876 -t lyktest -p hello
RocketMQLog:WARN No appenders could be found for logger (io.netty.util.internal.PlatformDependent0).
RocketMQLog:WARN Please initialize the logger system properly.
java.lang.RuntimeException: SendMessageCommand command failed
at org.apache.rocketmq.tools.command.message.SendMessageCommand.execute(SendMessageCommand.java:128)
at org.apache.rocketmq.tools.command.MQAdminStartup.main0(MQAdminStartup.java:140)
at org.apache.rocketmq.tools.command.MQAdminStartup.main(MQAdminStartup.java:91)
Caused by: org.apache.rocketmq.client.exception.MQClientException: Send [3] times, still failed, cost [207]ms, Topic: lyktest, BrokersSent: [RaftNode00, RaftNode00, RaftNode00]
See http://rocketmq.apache.org/docs/faq/ for further details.
at org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl.sendDefaultImpl(DefaultMQProducerImpl.java:665)
at org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl.send(DefaultMQProducerImpl.java:1343)
at org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl.send(DefaultMQProducerImpl.java:1289)
at org.apache.rocketmq.client.producer.DefaultMQProducer.send(DefaultMQProducer.java:325)
at org.apache.rocketmq.tools.command.message.SendMessageCommand.execute(SendMessageCommand.java:124)
... 2 more
Caused by: org.apache.rocketmq.client.exception.MQBrokerException: CODE: 1 DESC: java.lang.NullPointerException, org.apache.rocketmq.common.message.MessageExt.socketAddress2ByteBuffer(MessageExt.java:77) BROKER: rocketmq-broker-1.rocketmq-broker-service.rocketmq:10911
For more information, please visit the url, http://rocketmq.apache.org/docs/faq/
at org.apache.rocketmq.client.impl.MQClientAPIImpl.processSendResponse(MQClientAPIImpl.java:666)
at org.apache.rocketmq.client.impl.MQClientAPIImpl.sendMessageSync(MQClientAPIImpl.java:505)
at org.apache.rocketmq.client.impl.MQClientAPIImpl.sendMessage(MQClientAPIImpl.java:487)
at org.apache.rocketmq.client.impl.MQClientAPIImpl.sendMessage(MQClientAPIImpl.java:431)
at org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl.sendKernelImpl(DefaultMQProducerImpl.java:854)
at org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl.sendDefaultImpl(DefaultMQProducerImpl.java:584)
... 6 more
```
It can't work.
What happened in my cluster and how can i fix it? Or it's bug in 4.8.0?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] liuyk0109 commented on issue #2574: The 4.8.0 version of the dledger mode does not work in k8s
Posted by GitBox <gi...@apache.org>.
liuyk0109 commented on issue #2574:
URL: https://github.com/apache/rocketmq/issues/2574#issuecomment-758438095
@francisoliverlee v1.15.3
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] goudai commented on issue #2574: The 4.8.0 version of the dledger mode does not work in k8s
Posted by GitBox <gi...@apache.org>.
goudai commented on issue #2574:
URL: https://github.com/apache/rocketmq/issues/2574#issuecomment-784852988
集群状态正常，如下图所示
创建主题也正常
发消息是抛异常
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] goudai removed a comment on issue #2574: The 4.8.0 version of the dledger mode does not work in k8s
Posted by GitBox <gi...@apache.org>.
goudai removed a comment on issue #2574:
URL: https://github.com/apache/rocketmq/issues/2574#issuecomment-785014455
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] goudai removed a comment on issue #2574: The 4.8.0 version of the dledger mode does not work in k8s
Posted by GitBox <gi...@apache.org>.
goudai removed a comment on issue #2574:
URL: https://github.com/apache/rocketmq/issues/2574#issuecomment-784867081
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] goudai commented on issue #2574: The 4.8.0 version of the dledger mode does not work in k8s
Posted by GitBox <gi...@apache.org>.
goudai commented on issue #2574:
URL: https://github.com/apache/rocketmq/issues/2574#issuecomment-784867081
> > 我遇到了同样的问题。
>
> 是启动存在问题，还是跟上面另外一个同学一样，发送时出现异常？如果可以的话，麻烦发一段报错日志出来，另外确认下是不是IPv6 的环境？
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] goudai edited a comment on issue #2574: The 4.8.0 version of the dledger mode does not work in k8s
Posted by GitBox <gi...@apache.org>.
goudai edited a comment on issue #2574:
URL: https://github.com/apache/rocketmq/issues/2574#issuecomment-784868715
我猜测可能是ipv6导致的该问题，请问该如何配置让其监听IPV4
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] goudai commented on issue #2574: The 4.8.0 version of the dledger mode does not work in k8s
Posted by GitBox <gi...@apache.org>.
goudai commented on issue #2574:
URL: https://github.com/apache/rocketmq/issues/2574#issuecomment-785002144
是否有什么参数能够强制其监听0.0.0.0 或者监听IPV4
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] goudai commented on issue #2574: The 4.8.0 version of the dledger mode does not work in k8s
Posted by GitBox <gi...@apache.org>.
goudai commented on issue #2574:
URL: https://github.com/apache/rocketmq/issues/2574#issuecomment-784043614
我遇到了同样的问题。
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] francisoliverlee commented on issue #2574: The 4.8.0 version of the dledger mode does not work in k8s
Posted by GitBox <gi...@apache.org>.
francisoliverlee commented on issue #2574:
URL: https://github.com/apache/rocketmq/issues/2574#issuecomment-758426670
what's k8s version?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] goudai commented on issue #2574: The 4.8.0 version of the dledger mode does not work in k8s
Posted by GitBox <gi...@apache.org>.
goudai commented on issue #2574:
URL: https://github.com/apache/rocketmq/issues/2574#issuecomment-785014455
Exception in thread "main" org.apache.rocketmq.client.exception.MQClientException: Send [3] times, still failed, cost [1328]ms, Topic: test, BrokersSent: [testraft00, testraft00, testraft00]
See http://rocketmq.apache.org/docs/faq/ for further details.
at org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl.sendDefaultImpl(DefaultMQProducerImpl.java:665)
at org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl.send(DefaultMQProducerImpl.java:1343)
at org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl.send(DefaultMQProducerImpl.java:1289)
at org.apache.rocketmq.client.producer.DefaultMQProducer.send(DefaultMQProducer.java:325)
at com.yonghui.SyncProducer.main(SyncProducer.java:25)
Caused by: org.apache.rocketmq.client.exception.MQBrokerException: CODE: 1 DESC: [PC_SYNCHRONIZED]broker busy, start flow control for a while BROKER: 10.9.112.113:30911
For more information, please visit the url, http://rocketmq.apache.org/docs/faq/
at org.apache.rocketmq.client.impl.MQClientAPIImpl.processSendResponse(MQClientAPIImpl.java:666)
at org.apache.rocketmq.client.impl.MQClientAPIImpl.sendMessageSync(MQClientAPIImpl.java:505)
at org.apache.rocketmq.client.impl.MQClientAPIImpl.sendMessage(MQClientAPIImpl.java:487)
at org.apache.rocketmq.client.impl.MQClientAPIImpl.sendMessage(MQClientAPIImpl.java:431)
at org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl.sendKernelImpl(DefaultMQProducerImpl.java:854)
at org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl.sendDefaultImpl(DefaultMQProducerImpl.java:584)
... 4 more
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] duhenglucky edited a comment on issue #2574: The 4.8.0 version of the dledger mode does not work in k8s
Posted by GitBox <gi...@apache.org>.
duhenglucky edited a comment on issue #2574:
URL: https://github.com/apache/rocketmq/issues/2574#issuecomment-784658892
> 我遇到了同样的问题。
是启动存在问题，还是跟上面另外一个同学一样，发送时出现异常？如果可以的话，麻烦发一段报错日志出来，另外确认下是不是IPv6 的环境？
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] duhenglucky commented on issue #2574: The 4.8.0 version of the dledger mode does not work in k8s
Posted by GitBox <gi...@apache.org>.
duhenglucky commented on issue #2574:
URL: https://github.com/apache/rocketmq/issues/2574#issuecomment-784658892
> 我遇到了同样的问题。
是启动存在问题，还是跟上面另外一个同学一样，发送时出现异常？如果可以的话，麻烦发一段报错日志出来
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] vongosling closed issue #2574: The 4.8.0 version of the dledger mode does not work in k8s
Posted by GitBox <gi...@apache.org>.
vongosling closed issue #2574:
URL: https://github.com/apache/rocketmq/issues/2574
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] goudai commented on issue #2574: The 4.8.0 version of the dledger mode does not work in k8s
Posted by GitBox <gi...@apache.org>.
goudai commented on issue #2574:
URL: https://github.com/apache/rocketmq/issues/2574#issuecomment-784868715
我猜测可能是ipv6导致的该问题
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org