You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@rocketmq.apache.org by GitBox <gi...@apache.org> on 2020/08/18 08:35:12 UTC

[GitHub] [rocketmq] dingyaK opened a new issue #2255: rocketmq的broker,没有不监听11011端口

dingyaK opened a new issue #2255:
URL: https://github.com/apache/rocketmq/issues/2255


   ![Uploading image.png…]()
   broker_default.log中出现error:too many open file 。现在,rocketmq没有监听11011端口。我们是2m-2s模式,互为主从。这台是broker-a服务器,目前造成的情况是:rocketmq在nameSrv中注册了,但是,没有监听11011端口,导致,生产着一直朝它生产消息,一直失败。报错是connect to<10.100.12.135:11011> failed (topic没有配置整个集群处理,仅仅配置了一台broker)。查询服务器就是截图的错误,没有监听11011端口。11015端口,broker-b的是从服务器


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [rocketmq] dingyaK commented on issue #2255: There broker in rocketmq that does not listen to port 11011

Posted by GitBox <gi...@apache.org>.
dingyaK commented on issue #2255:
URL: https://github.com/apache/rocketmq/issues/2255#issuecomment-675834805


   Can you help me see what is wrong with the top command


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [rocketmq] imaffe commented on issue #2255: There broker in rocketmq that does not listen to port 11011

Posted by GitBox <gi...@apache.org>.
imaffe commented on issue #2255:
URL: https://github.com/apache/rocketmq/issues/2255#issuecomment-675832625


   > rocketmq version 4.3.0
   
   yeah, so here are some suggestions to debug this.
   
   1) Since there is too many files exceptions, I think we might trust it until we find further evidence. The accepter thread might not be robust enough to handle too many file open exception for version 4.3.0.  This might be the most likely cause
   
   
   2) Upgrade to the newest rocketMQ version, as 4.3.0 is too old and might have some unexpected bugs.
   
   
   Hope these helps.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [rocketmq] dingyaK commented on issue #2255: There broker in rocketmq that does not listen to port 11011

Posted by GitBox <gi...@apache.org>.
dingyaK commented on issue #2255:
URL: https://github.com/apache/rocketmq/issues/2255#issuecomment-675823152


   tcp6 0 0 10.100.12.135:50974 10.100.12.136:9876 ESTABLISHED 12217/java
   tcp6 0 0 10.100.12.135:47132 10.100.12.135:9876 ESTABLISHED 12217/java
   
   These two records indicate that broker-a registers routing information with namesrv. However, broker-a is no longer available at this time, so the producer produces messages to this broker-a, so it fails. The reason for the failure is that port 11011 is not monitored, why suddenly port 11011 is not monitored?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [rocketmq] imaffe commented on issue #2255: There broker in rocketmq that does not listen to port 11011

Posted by GitBox <gi...@apache.org>.
imaffe commented on issue #2255:
URL: https://github.com/apache/rocketmq/issues/2255#issuecomment-675826383


   Another problem might just be (just a guess) your Linux user are not allowed to open a lot of sockets, and your connections has exceeded that amount, then RocketMQ crashed on this error or something and fail to serve again. Maybe you want to validate if it is really caused by too many files open crashed the accepting thread first ~


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [rocketmq] imaffe commented on issue #2255: There broker in rocketmq that does not listen to port 11011

Posted by GitBox <gi...@apache.org>.
imaffe commented on issue #2255:
URL: https://github.com/apache/rocketmq/issues/2255#issuecomment-675869274


   Sorry, but all I can provide is a direction. As for how to verify if that caused the problem, or debugging the problem, I'm afraid it's beyond my capability. I would be happy to follow this issue up if you find out it's not "too many open files" problem  and upgraded to 4.7.1. 
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [rocketmq] imaffe commented on issue #2255: There broker in rocketmq that does not listen to port 11011

Posted by GitBox <gi...@apache.org>.
imaffe commented on issue #2255:
URL: https://github.com/apache/rocketmq/issues/2255#issuecomment-675813221


   Broker is listening on 10911 by default, curious when does broker need to listen on port 11011 and why client tries to connect to port 11011. Did you change to broker to listen on 11011 ?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [rocketmq] dingyaK commented on issue #2255: There broker in rocketmq that does not listen to port 11011

Posted by GitBox <gi...@apache.org>.
dingyaK commented on issue #2255:
URL: https://github.com/apache/rocketmq/issues/2255#issuecomment-675834008


   [新建文本文档.txt](https://github.com/apache/rocketmq/files/5093896/default.txt)
   This is too many open file abnormal file. 
   [psef.txt](https://github.com/apache/rocketmq/files/5093898/psef.txt)
   This is all the tasks of the current machine
   top - 11:34:46 up 32 days,  9:49,  1 user,  load average: 0.04, 0.01, 0.00
   Tasks: 132 total,   1 running,  70 sleeping,   0 stopped,   0 zombie
   %Cpu(s):  0.3 us,  0.3 sy,  0.0 ni, 99.0 id,  0.0 wa,  0.0 hi,  0.5 si,  0.0 st
   MiB Mem :   7625.1 total,   1802.5 free,   4327.0 used,   1495.5 buff/cache
   MiB Swap:      0.0 total,      0.0 free,      0.0 used.   2554.3 avail Mem 
   
     PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                                                                                                                                                  
   12217 root      20   0 7403.1m   1.3g  16.8m S   0.7 17.9   2004:40 java                                                                                                                                                                                                                                                     
   12365 root      20   0 7328.3m   1.2g  20.3m S   0.3 15.9 260:57.90 java                                                                                                                                                                                                                                                     
   13748 root      20   0 5208.4m 907.5m  10.2m S   0.3 11.9 108:22.23 java                                                                                                                                                                                                                                                     
     933 mq        20   0 4689.4m 540.6m   3.0m S   0.0  7.1  45:18.83 java          


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [rocketmq] dingyaK commented on issue #2255: There broker in rocketmq that does not listen to port 11011

Posted by GitBox <gi...@apache.org>.
dingyaK commented on issue #2255:
URL: https://github.com/apache/rocketmq/issues/2255#issuecomment-675894704


   @imaffe Thanks a lot


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [rocketmq] dingyaK commented on issue #2255: There broker in rocketmq that does not listen to port 11011

Posted by GitBox <gi...@apache.org>.
dingyaK commented on issue #2255:
URL: https://github.com/apache/rocketmq/issues/2255#issuecomment-675817482


   tcp6       0      0 :::11009                :::*                    LISTEN      2351/java           
   tcp6       0      0 :::11011                :::*                    LISTEN      2351/java           
   tcp6       0      0 :::11012                :::*                    LISTEN      2351/java           
   tcp6       0      0 10.100.12.136:11011     10.100.11.177:54706     ESTABLISHED 2351/java           
   tcp6       0      0 10.100.12.136:11011     10.100.11.177:36194     ESTABLISHED 2351/java           
   tcp6       0      0 10.100.12.136:11009     10.100.12.135:41346     ESTABLISHED 2351/java           
   tcp6       0      0 10.100.12.136:11011     10.100.12.6:44810       ESTABLISHED 2351/java           
   tcp6       0      0 10.100.12.136:11011     10.100.11.177:14383     ESTABLISHED 2351/java           
   tcp6       0      0 172.17.0.1:11012        10.100.12.136:46826     ESTABLISHED 2351/java           
   tcp6       0      0 10.100.12.136:11009     10.100.11.177:44227     ESTABLISHED 2351/java           
   tcp6       0      0 10.100.12.136:11011     10.100.11.106:54258     ESTABLISHED 2351/java           
   tcp6       0      0 10.100.12.136:11009     10.100.11.177:17580     ESTABLISHED 2351/java           
   tcp6       0      0 10.100.12.136:11011     10.100.13.32:3201       ESTABLISHED 2351/java           
   tcp6       0      0 10.100.12.136:11011     10.100.11.106:43768     ESTABLISHED 2351/java           
   tcp6       0      0 10.100.12.136:11011     10.100.11.39:30543      ESTABLISHED 2351/java           
   tcp6       0      0 10.100.12.136:11011     10.100.11.39:36251      ESTABLISHED 2351/java           
   tcp6       0      0 10.100.12.136:11011     10.100.11.244:23251     ESTABLISHED 2351/java           
   tcp6       0      0 10.100.12.136:11009     10.100.11.106:52049     ESTABLISHED 2351/java           
   tcp6       0      0 10.100.12.136:11011     10.100.11.177:46674     ESTABLISHED 2351/java           
   tcp6       0      0 10.100.12.136:11009     10.100.11.177:15315     ESTABLISHED 2351/java           
   tcp6       0      0 10.100.12.136:11011     10.100.13.32:60807      ESTABLISHED 2351/java           
   tcp6       0      0 10.100.12.136:11009     10.100.11.177:52870     ESTABLISHED 2351/java           
   tcp6       0      0 10.100.12.136:11009     10.100.13.32:41858      ESTABLISHED 2351/java           
   tcp6       0      0 10.100.12.136:37520     10.100.12.135:9876      ESTABLISHED 2351/java           
   tcp6       0      0 10.100.12.136:11011     10.100.12.135:57948     ESTABLISHED 2351/java           
   tcp6       0      0 10.100.12.136:11009     10.100.11.39:47653      ESTABLISHED 2351/java           
   tcp6       0      0 10.100.12.136:11009     10.100.11.39:53339      ESTABLISHED 2351/java           
   tcp6       0      0 10.100.12.136:11009     10.100.11.39:38939      ESTABLISHED 2351/java           
   tcp6       0      0 10.100.12.136:45042     10.100.12.136:9876      ESTABLISHED 2351/java           
   tcp6       0      0 10.100.12.136:11011     10.100.11.39:57901      ESTABLISHED 2351/java           
   unix  2      [ ]         STREAM     CONNECTED     23532    2351/java            
   unix  2      [ ]         STREAM     CONNECTED     27114    2351/java    
   
   This is another broker-b-master, which has always been good and has been listening on 3 ports. However, the broker-a-master just started, only listening on one port after starting it for a while


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [rocketmq] dingyaK commented on issue #2255: There broker in rocketmq that does not listen to port 11011

Posted by GitBox <gi...@apache.org>.
dingyaK commented on issue #2255:
URL: https://github.com/apache/rocketmq/issues/2255#issuecomment-675815765


   @imaffe It's fixed every time you restart, but it's going to happen again later


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [rocketmq] imaffe commented on issue #2255: There broker in rocketmq that does not listen to port 11011

Posted by GitBox <gi...@apache.org>.
imaffe commented on issue #2255:
URL: https://github.com/apache/rocketmq/issues/2255#issuecomment-675824927


   Can you provide which version rocketmq you are using, what environment and any other information you might think useful ? It's weird because afaik broker should not listen on 11011 or 11009. I might need to search in the source code to find out why your producer is sending to broker on port 11011.
   
   Also on my machine it's
   ![image](https://user-images.githubusercontent.com/22848576/90587561-44348900-e20c-11ea-9980-9dfda5088667.png)
   
   So is it possible you are using an old version of rocketmq and this bug might already been fixed?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [rocketmq] imaffe closed issue #2255: There broker in rocketmq that does not listen to port 11011

Posted by GitBox <gi...@apache.org>.
imaffe closed issue #2255:
URL: https://github.com/apache/rocketmq/issues/2255


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [rocketmq] dingyaK commented on issue #2255: There broker in rocketmq that does not listen to port 11011

Posted by GitBox <gi...@apache.org>.
dingyaK commented on issue #2255:
URL: https://github.com/apache/rocketmq/issues/2255#issuecomment-675815253


   The default port of rocketmq has not been changed. Suddenly, port 11011 is not monitored,


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [rocketmq] dingyaK edited a comment on issue #2255: There broker in rocketmq that does not listen to port 11011

Posted by GitBox <gi...@apache.org>.
dingyaK edited a comment on issue #2255:
URL: https://github.com/apache/rocketmq/issues/2255#issuecomment-675834805


   psef is ps -aux command result 
   Can you help me see what is wrong with the top command


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [rocketmq] dingyaK commented on issue #2255: There broker in rocketmq that does not listen to port 11011

Posted by GitBox <gi...@apache.org>.
dingyaK commented on issue #2255:
URL: https://github.com/apache/rocketmq/issues/2255#issuecomment-675831593


   rocketmq version 4.3.0


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [rocketmq] dingyaK commented on issue #2255: There broker in rocketmq that does not listen to port 11011

Posted by GitBox <gi...@apache.org>.
dingyaK commented on issue #2255:
URL: https://github.com/apache/rocketmq/issues/2255#issuecomment-675831498


   At the beginning, I also thought that there were too many handles, but the ls /proc/pid/fd |wc -l command only had more than 300. It may have reached 4096, then released. Currently, broker_default.log only has too many open file exceptions
   
   The following file is the file of broker_default.log and fd, the pid may be different because of restart
   [1(3).txt](https://github.com/apache/rocketmq/files/5093862/1.3.txt)
   [1(2).txt](https://github.com/apache/rocketmq/files/5093863/1.2.txt)
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org