You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Guillaume Cripiau <gu...@murprogroup.com> on 2021/11/02 11:19:18 UTC

Connection reset / Already connected / Not connected

Hello,

We need support. Many devices disconnect for no apparent reason and 
cannot reconnect after several days. Could you help us solve this 
problem? Here is an example in the logs :

> 2021-11-02 10:30:44,413 | WARN  | Transport Connection to: 
> tcp://62.83.10.35:1024 failed | 
> org.apache.activemq.broker.TransportConnection.Transport | Async 
> Exception Handler | localhost            | |
> java.io.IOException: Connection reset
>     at 
> org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:46)[activemq-client-5.16.3.jar:5.16.3]
>     at 
> org.apache.activemq.transport.mqtt.MQTTTransportFilter.oneway(MQTTTransportFilter.java:83)[activemq-mqtt-5.16.3.jar:5.16.3]
>     at 
> org.apache.activemq.transport.mqtt.MQTTInactivityMonitor.doOnewaySend(MQTTInactivityMonitor.java:185)[activemq-mqtt-5.16.3.jar:5.16.3]
>     at 
> org.apache.activemq.transport.mqtt.MQTTInactivityMonitor.oneway(MQTTInactivityMonitor.java:174)[activemq-mqtt-5.16.3.jar:5.16.3]
>     at 
> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)[activemq-client-5.16.3.jar:5.16.3]
>     at 
> org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1485)[activemq-broker-5.16.3.jar:5.16.3]
>     at 
> org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:982)[activemq-broker-5.16.3.jar:5.16.3]
>     at 
> org.apache.activemq.broker.TransportConnection.iterate(TransportConnection.java:1033)[activemq-broker-5.16.3.jar:5.16.3]
>     at 
> org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)[activemq-client-5.16.3.jar:5.16.3]
>     at 
> org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)[activemq-client-5.16.3.jar:5.16.3]
>     at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown 
> Source)[:1.8.0_211]
>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown 
> Source)[:1.8.0_211]
>     at java.lang.Thread.run(Unknown Source)[:1.8.0_211]
> 2021-11-02 10:30:46,359 | WARN  | Transport Connection to: 
> tcp://62.83.10.35:35598 failed | 
> org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ 
> Transport: tcp:///62.83.10.35:35598@1883 | localhost            | |
> org.apache.activemq.transport.mqtt.MQTTProtocolException: Already 
> connected.
>     at 
> org.apache.activemq.transport.mqtt.MQTTProtocolConverter.onMQTTConnect(MQTTProtocolConverter.java:239)[activemq-mqtt-5.16.3.jar:5.16.3]
>     at 
> org.apache.activemq.transport.mqtt.MQTTProtocolConverter.onMQTTCommand(MQTTProtocolConverter.java:204)[activemq-mqtt-5.16.3.jar:5.16.3]
>     at 
> org.apache.activemq.transport.mqtt.MQTTTransportFilter.onCommand(MQTTTransportFilter.java:94)[activemq-mqtt-5.16.3.jar:5.16.3]
>     at 
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)[activemq-client-5.16.3.jar:5.16.3]
>     at 
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:233)[activemq-client-5.16.3.jar:5.16.3]
>     at 
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215)[activemq-client-5.16.3.jar:5.16.3]
>     at java.lang.Thread.run(Unknown Source)[:1.8.0_211]
> 2021-11-02 10:31:19,226 | WARN  | Transport Connection to: 
> tcp://62.83.10.35:3000 failed | 
> org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ 
> Transport: tcp:///62.83.10.35:3000@1883 | localhost            | |
> org.apache.activemq.transport.mqtt.MQTTProtocolException: Not connected.
>     at 
> org.apache.activemq.transport.mqtt.MQTTProtocolConverter.checkConnected(MQTTProtocolConverter.java:736)[activemq-mqtt-5.16.3.jar:5.16.3]
>     at 
> org.apache.activemq.transport.mqtt.MQTTProtocolConverter.onSubscribe(MQTTProtocolConverter.java:377)[activemq-mqtt-5.16.3.jar:5.16.3]
>     at 
> org.apache.activemq.transport.mqtt.MQTTProtocolConverter.onMQTTCommand(MQTTProtocolConverter.java:212)[activemq-mqtt-5.16.3.jar:5.16.3]
>     at 
> org.apache.activemq.transport.mqtt.MQTTTransportFilter.onCommand(MQTTTransportFilter.java:94)[activemq-mqtt-5.16.3.jar:5.16.3]
>     at 
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)[activemq-client-5.16.3.jar:5.16.3]
>     at 
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:233)[activemq-client-5.16.3.jar:5.16.3]
>     at 
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215)[activemq-client-5.16.3.jar:5.16.3]
>     at java.lang.Thread.run(Unknown Source)[:1.8.0_211]

Regards

Guillaume Cripiau

Re: Connection reset / Already connected / Not connected

Posted by Tim Bain <tb...@alumni.duke.edu>.
When LDAP auth is enabled and this problem is occurring, could you please
use VisualVM or another performance analyzer to capture a CPU sampling
snapshot over a minute or so? That should allow us to understand what's so
slow about the LDAP code path.

Thanks,
Tim

On Tue, Jan 25, 2022, 8:44 AM Guillaume Cripiau <
guillaume.cripiau@murprogroup.com> wrote:

> Hello,
>
> Apache LDAP is used to give access to topic. When Apache LDAP is disable,
> the device connect normally. When Apache LDAP is enable, the latency can
> exceed one second and the devices have problems to connect because of their
> timeout. We don't have this latency with another message broker.
>
> Here are CPU and memory load in attachment.
>
> We sometimes have centreon alert. See connection JMX captures in
> attachment :
> - CTA_2022-01-19_06-31.png  -> 4 cores at 100%. CPU for activemq is at 389%
> - CTA_2022-01-19_06-33.png -> after the reboot
>
> ActiveMQ config :
>
> *VM Summary* mercredi 19 janvier 2022 8 h 36 min 48 s CET
> ------------------------------
> Connection name:
> admin@service:jmx:rmi:///jndi/rmi://51.255.198.17:31616/jmxrmi
> Virtual Machine:  OpenJDK 64-Bit Server VM version 25.312-b07
> Vendor:  Red Hat, Inc.
> Name:  11604@vps391568.ovh.net
> Uptime:  32 days 19 hours 56 minutes
> Process CPU time:  1 hour 54 minutes
> JIT compiler:  HotSpot 64-Bit Tiered Compilers
> Total compile time:  1 minute
> ------------------------------
> Live threads:     59
> Peak:     70
> Daemon threads:     37
> Total threads started:  7.070
> Current classes loaded:  8.584
> Total classes loaded:  8.588
> Total classes unloaded:      4
>
>
> ------------------------------
> Current heap size:    565.627 kbytes
> Maximum heap size:  1.864.192 kbytes
> Committed memory:  953.856 kbytes
> Pending finalization:  0 objects
> Garbage collector:  Name = 'PS MarkSweep', Collections = 2, Total time
> spent = 0,180 seconds
> Garbage collector:  Name = 'PS Scavenge', Collections = 183, Total time
> spent = 47,516 seconds
> ------------------------------
> Operating System:  Linux 3.10.0-957.21.3.el7.x86_64
> Architecture:  amd64
> Number of processors:  2
> Committed virtual memory:  4.782.976 kbytes
> Total physical memory:  7.814.608 kbytes
> Free physical memory:  3.406.084 kbytes
> Total swap space:          0 kbytes
> Free swap space:          0 kbytes
> ------------------------------
> VM arguments:  -Xms1024M -Xmx2G
> -Djava.util.logging.config.file=logging.properties
> -Djava.security.auth.login.config=/opt/apache-activemq-5.16.3//conf/login.config
> -Dorg.apache.activemq.audit=true -Djava.rmi.server.hostname=51.255.198.17
> -Dcom.sun.management.jmxremote.port=31616
> -Dcom.sun.management.jmxremote.password.file=/opt/apache-activemq-5.16.3//conf/jmx.password
> -Dcom.sun.management.jmxremote.access.file=/opt/apache-activemq-5.16.3//conf/jmx.access
> -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote
> -Djava.awt.headless=true -Djava.io.tmpdir=/opt/apache-activemq-5.16.3//tmp
> -Dactivemq.classpath=/opt/apache-activemq-5.16.3//
> conf:/opt/apache-activemq-5.16.3//../lib/:
> -Dactivemq.home=/opt/apache-activemq-5.16.3/
> -Dactivemq.base=/opt/apache-activemq-5.16.3/
> -Dactivemq.conf=/opt/apache-activemq-5.16.3//conf
> -Dactivemq.data=/opt/apache-activemq-5.16.3//data
> Class path:  /opt/apache-activemq-5.16.3//bin/activemq.jar
> Library path:
> /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> Boot class path:
> /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre/lib/resources.jar:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre/lib/rt.jar:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre/lib/jsse.jar:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre/lib/jce.jar:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre/lib/charsets.jar:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre/lib/jfr.jar:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre/classes
>
> Regards
>
> Guillaume Cripiau
>
> Le 15-11-21 à 14:10, Tim Bain a écrit :
>
> What is the CPU load like on the broker host? Is that process spiking the
> CPU? If so, can you use JVisualVM or similar to do CPU sampling of the
> broker process and find out where it's spending its time?
>
> My suspicion is that the broker is so busy that it's not responding to some
> connections fast enough to let them succeed.
>
> Tim
>
> On Wed, Nov 10, 2021, 2:17 AM Guillaume Cripiau <gu...@murprogroup.com> wrote:
>
>
> Hello,
>
> The only other logs (mqtt messages on the server) show this :
> 10:30:45
> remove/ID:airprotec-cta-41267-1635489094200-3:8407/03200646/Connection
> 10:30:46
> add/ID:airprotec-cta-41267-1635489094200-3:8410/03200646/Connection
> 10:30:46
> remove/ID:airprotec-cta-41267-1635489094200-3:8410/03200646/Connection
>
> Yes, the client restarts after 2 hours without connection and the same
> event happens but not always.
>
> The clients try to reconnect, the broker has just restarted.
>
> We cannot reproduce the problem on our dev server with minimal setup.
>
> We actually analyse the TCP connexion : the clients use the source port
> 1883 and the timeout between a CONNECT command and CONNACK is 1 second.
> Can this settings be the cause of our problems?
>
> Regards
>
> Guillaume Cripiau
>
> Le 02-11-21 à 12:45, Tim Bain a écrit :
>
> These are broker logs, or client logs? Whichever it is, what's in the
>
> other
>
> process's logs at the same time?
>
> You said that reconnecting isn't possible for hours after this happens.
>
> Do
>
> you see the same messages in the logs for that whole time?
>
> At the time this happens, what are the client(s) and the broker doing?
>
> Can you reproduce the problem reliably in a minimal setup (just one or
>
> two
>
> clients, minimal time and message load)?
>
> Tim
>
> On Tue, Nov 2, 2021, 5:19 AM Guillaume Cripiau <gu...@murprogroup.com> wrote:
>
>
> Hello,
>
> We need support. Many devices disconnect for no apparent reason and
> cannot reconnect after several days. Could you help us solve this
> problem? Here is an example in the logs :
>
>
> 2021-11-02 10:30:44,413 | WARN  | Transport Connection to:
> tcp://62.83.10.35:1024 failed |
> org.apache.activemq.broker.TransportConnection.Transport | Async
> Exception Handler | localhost            | |
> java.io.IOException: Connection reset
>      at
>
>
>  org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:46)[activemq-client-5.16.3.jar:5.16.3]
>
>      at
>
>
>  org.apache.activemq.transport.mqtt.MQTTTransportFilter.oneway(MQTTTransportFilter.java:83)[activemq-mqtt-5.16.3.jar:5.16.3]
>
>      at
>
>
>  org.apache.activemq.transport.mqtt.MQTTInactivityMonitor.doOnewaySend(MQTTInactivityMonitor.java:185)[activemq-mqtt-5.16.3.jar:5.16.3]
>
>      at
>
>
>  org.apache.activemq.transport.mqtt.MQTTInactivityMonitor.oneway(MQTTInactivityMonitor.java:174)[activemq-mqtt-5.16.3.jar:5.16.3]
>
>      at
>
>
>  org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)[activemq-client-5.16.3.jar:5.16.3]
>
>      at
>
>
>  org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1485)[activemq-broker-5.16.3.jar:5.16.3]
>
>      at
>
>
>  org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:982)[activemq-broker-5.16.3.jar:5.16.3]
>
>      at
>
>
>  org.apache.activemq.broker.TransportConnection.iterate(TransportConnection.java:1033)[activemq-broker-5.16.3.jar:5.16.3]
>
>      at
>
>
>  org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)[activemq-client-5.16.3.jar:5.16.3]
>
>      at
>
>
>  org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)[activemq-client-5.16.3.jar:5.16.3]
>
>      at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
> Source)[:1.8.0_211]
>      at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source)[:1.8.0_211]
>      at java.lang.Thread.run(Unknown Source)[:1.8.0_211]
> 2021-11-02 10:30:46,359 | WARN  | Transport Connection to:
> tcp://62.83.10.35:35598 failed |
> org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ
> Transport: tcp:///62.83.10.35:35598@1883 | localhost            | |
> org.apache.activemq.transport.mqtt.MQTTProtocolException: Already
> connected.
>      at
>
>
>  org.apache.activemq.transport.mqtt.MQTTProtocolConverter.onMQTTConnect(MQTTProtocolConverter.java:239)[activemq-mqtt-5.16.3.jar:5.16.3]
>
>      at
>
>
>  org.apache.activemq.transport.mqtt.MQTTProtocolConverter.onMQTTCommand(MQTTProtocolConverter.java:204)[activemq-mqtt-5.16.3.jar:5.16.3]
>
>      at
>
>
>  org.apache.activemq.transport.mqtt.MQTTTransportFilter.onCommand(MQTTTransportFilter.java:94)[activemq-mqtt-5.16.3.jar:5.16.3]
>
>      at
>
>
>  org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)[activemq-client-5.16.3.jar:5.16.3]
>
>      at
>
>
>  org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:233)[activemq-client-5.16.3.jar:5.16.3]
>
>      at
>
>
>  org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215)[activemq-client-5.16.3.jar:5.16.3]
>
>      at java.lang.Thread.run(Unknown Source)[:1.8.0_211]
> 2021-11-02 10:31:19,226 | WARN  | Transport Connection to:
> tcp://62.83.10.35:3000 failed |
> org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ
> Transport: tcp:///62.83.10.35:3000@1883 | localhost            | |
> org.apache.activemq.transport.mqtt.MQTTProtocolException: Not
>
> connected.
>
>      at
>
>
>  org.apache.activemq.transport.mqtt.MQTTProtocolConverter.checkConnected(MQTTProtocolConverter.java:736)[activemq-mqtt-5.16.3.jar:5.16.3]
>
>      at
>
>
>  org.apache.activemq.transport.mqtt.MQTTProtocolConverter.onSubscribe(MQTTProtocolConverter.java:377)[activemq-mqtt-5.16.3.jar:5.16.3]
>
>      at
>
>
>  org.apache.activemq.transport.mqtt.MQTTProtocolConverter.onMQTTCommand(MQTTProtocolConverter.java:212)[activemq-mqtt-5.16.3.jar:5.16.3]
>
>      at
>
>
>  org.apache.activemq.transport.mqtt.MQTTTransportFilter.onCommand(MQTTTransportFilter.java:94)[activemq-mqtt-5.16.3.jar:5.16.3]
>
>      at
>
>
>  org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)[activemq-client-5.16.3.jar:5.16.3]
>
>      at
>
>
>  org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:233)[activemq-client-5.16.3.jar:5.16.3]
>
>      at
>
>
>  org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215)[activemq-client-5.16.3.jar:5.16.3]
>
>      at java.lang.Thread.run(Unknown Source)[:1.8.0_211]
>
> Regards
>
> Guillaume Cripiau
>
>
>

Re: Connection reset / Already connected / Not connected

Posted by Guillaume Cripiau <gu...@murprogroup.com>.
Hello,

Apache LDAP is used to give access to topic. When Apache LDAP is 
disable, the device connect normally. When Apache LDAP is enable, the 
latency can exceed one second and the devices have problems to connect 
because of their timeout. We don't have this latency with another 
message broker.

Here are CPU and memory load in attachment.

We sometimes have centreon alert. See connection JMX captures in 
attachment :
- CTA_2022-01-19_06-31.png  -> 4 cores at 100%. CPU for activemq is at 389%
- CTA_2022-01-19_06-33.png -> after the reboot

ActiveMQ config :

*VM Summary*

mercredi 19 janvier 2022 8 h 36 min 48 s CET

------------------------------------------------------------------------
Connection name: 
admin@service:jmx:rmi:///jndi/rmi://51.255.198.17:31616/jmxrmi
Virtual Machine: 	OpenJDK 64-Bit Server VM version 25.312-b07
Vendor: 	Red Hat, Inc.
Name: 	11604@vps391568.ovh.net

	
Uptime: 	32 days 19 hours 56 minutes
Process CPU time: 	1 hour 54 minutes
JIT compiler: 	HotSpot 64-Bit Tiered Compilers
Total compile time: 	1 minute

------------------------------------------------------------------------
Live threads: 	   59
Peak: 	   70
Daemon threads: 	   37
Total threads started: 	7.070

	
Current classes loaded: 	8.584
Total classes loaded: 	8.588
Total classes unloaded: 	    4

	

------------------------------------------------------------------------
Current heap size: 	  565.627 kbytes
Maximum heap size: 	1.864.192 kbytes

	
Committed memory: 	953.856 kbytes
Pending finalization: 	0 objects

Garbage collector: 	Name = 'PS MarkSweep', Collections = 2, Total time 
spent = 0,180 seconds
Garbage collector: 	Name = 'PS Scavenge', Collections = 183, Total time 
spent = 47,516 seconds

------------------------------------------------------------------------
Operating System: 	Linux 3.10.0-957.21.3.el7.x86_64
Architecture: 	amd64
Number of processors: 	2
Committed virtual memory: 	4.782.976 kbytes

	
Total physical memory: 	7.814.608 kbytes
Free physical memory: 	3.406.084 kbytes
Total swap space: 	        0 kbytes
Free swap space: 	        0 kbytes

------------------------------------------------------------------------
VM arguments: 	-Xms1024M -Xmx2G 
-Djava.util.logging.config.file=logging.properties 
-Djava.security.auth.login.config=/opt/apache-activemq-5.16.3//conf/login.config 
-Dorg.apache.activemq.audit=true 
-Djava.rmi.server.hostname=51.255.198.17 
-Dcom.sun.management.jmxremote.port=31616 
-Dcom.sun.management.jmxremote.password.file=/opt/apache-activemq-5.16.3//conf/jmx.password 
-Dcom.sun.management.jmxremote.access.file=/opt/apache-activemq-5.16.3//conf/jmx.access 
-Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote 
-Djava.awt.headless=true 
-Djava.io.tmpdir=/opt/apache-activemq-5.16.3//tmp 
-Dactivemq.classpath=/opt/apache-activemq-5.16.3//conf:/opt/apache-activemq-5.16.3//../lib/: 
-Dactivemq.home=/opt/apache-activemq-5.16.3/ 
-Dactivemq.base=/opt/apache-activemq-5.16.3/ 
-Dactivemq.conf=/opt/apache-activemq-5.16.3//conf 
-Dactivemq.data=/opt/apache-activemq-5.16.3//data
Class path: 	/opt/apache-activemq-5.16.3//bin/activemq.jar
Library path: 	/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
Boot class path: 
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre/lib/resources.jar:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre/lib/rt.jar:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre/lib/jsse.jar:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre/lib/jce.jar:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre/lib/charsets.jar:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre/lib/jfr.jar:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre/classes 



Regards

Guillaume Cripiau

Le 15-11-21 à 14:10, Tim Bain a écrit :
> What is the CPU load like on the broker host? Is that process spiking the
> CPU? If so, can you use JVisualVM or similar to do CPU sampling of the
> broker process and find out where it's spending its time?
>
> My suspicion is that the broker is so busy that it's not responding to some
> connections fast enough to let them succeed.
>
> Tim
>
> On Wed, Nov 10, 2021, 2:17 AM Guillaume Cripiau <
> guillaume.cripiau@murprogroup.com> wrote:
>
>> Hello,
>>
>> The only other logs (mqtt messages on the server) show this :
>> 10:30:45
>> remove/ID:airprotec-cta-41267-1635489094200-3:8407/03200646/Connection
>> 10:30:46
>> add/ID:airprotec-cta-41267-1635489094200-3:8410/03200646/Connection
>> 10:30:46
>> remove/ID:airprotec-cta-41267-1635489094200-3:8410/03200646/Connection
>>
>> Yes, the client restarts after 2 hours without connection and the same
>> event happens but not always.
>>
>> The clients try to reconnect, the broker has just restarted.
>>
>> We cannot reproduce the problem on our dev server with minimal setup.
>>
>> We actually analyse the TCP connexion : the clients use the source port
>> 1883 and the timeout between a CONNECT command and CONNACK is 1 second.
>> Can this settings be the cause of our problems?
>>
>> Regards
>>
>> Guillaume Cripiau
>>
>> Le 02-11-21 à 12:45, Tim Bain a écrit :
>>> These are broker logs, or client logs? Whichever it is, what's in the
>> other
>>> process's logs at the same time?
>>>
>>> You said that reconnecting isn't possible for hours after this happens.
>> Do
>>> you see the same messages in the logs for that whole time?
>>>
>>> At the time this happens, what are the client(s) and the broker doing?
>>>
>>> Can you reproduce the problem reliably in a minimal setup (just one or
>> two
>>> clients, minimal time and message load)?
>>>
>>> Tim
>>>
>>> On Tue, Nov 2, 2021, 5:19 AM Guillaume Cripiau <
>>> guillaume.cripiau@murprogroup.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> We need support. Many devices disconnect for no apparent reason and
>>>> cannot reconnect after several days. Could you help us solve this
>>>> problem? Here is an example in the logs :
>>>>
>>>>> 2021-11-02 10:30:44,413 | WARN  | Transport Connection to:
>>>>> tcp://62.83.10.35:1024 failed |
>>>>> org.apache.activemq.broker.TransportConnection.Transport | Async
>>>>> Exception Handler | localhost            | |
>>>>> java.io.IOException: Connection reset
>>>>>       at
>>>>>
>> org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:46)[activemq-client-5.16.3.jar:5.16.3]
>>>>>       at
>>>>>
>> org.apache.activemq.transport.mqtt.MQTTTransportFilter.oneway(MQTTTransportFilter.java:83)[activemq-mqtt-5.16.3.jar:5.16.3]
>>>>>       at
>>>>>
>> org.apache.activemq.transport.mqtt.MQTTInactivityMonitor.doOnewaySend(MQTTInactivityMonitor.java:185)[activemq-mqtt-5.16.3.jar:5.16.3]
>>>>>       at
>>>>>
>> org.apache.activemq.transport.mqtt.MQTTInactivityMonitor.oneway(MQTTInactivityMonitor.java:174)[activemq-mqtt-5.16.3.jar:5.16.3]
>>>>>       at
>>>>>
>> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)[activemq-client-5.16.3.jar:5.16.3]
>>>>>       at
>>>>>
>> org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1485)[activemq-broker-5.16.3.jar:5.16.3]
>>>>>       at
>>>>>
>> org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:982)[activemq-broker-5.16.3.jar:5.16.3]
>>>>>       at
>>>>>
>> org.apache.activemq.broker.TransportConnection.iterate(TransportConnection.java:1033)[activemq-broker-5.16.3.jar:5.16.3]
>>>>>       at
>>>>>
>> org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)[activemq-client-5.16.3.jar:5.16.3]
>>>>>       at
>>>>>
>> org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)[activemq-client-5.16.3.jar:5.16.3]
>>>>>       at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
>>>>> Source)[:1.8.0_211]
>>>>>       at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
>>>>> Source)[:1.8.0_211]
>>>>>       at java.lang.Thread.run(Unknown Source)[:1.8.0_211]
>>>>> 2021-11-02 10:30:46,359 | WARN  | Transport Connection to:
>>>>> tcp://62.83.10.35:35598 failed |
>>>>> org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ
>>>>> Transport: tcp:///62.83.10.35:35598@1883 | localhost            | |
>>>>> org.apache.activemq.transport.mqtt.MQTTProtocolException: Already
>>>>> connected.
>>>>>       at
>>>>>
>> org.apache.activemq.transport.mqtt.MQTTProtocolConverter.onMQTTConnect(MQTTProtocolConverter.java:239)[activemq-mqtt-5.16.3.jar:5.16.3]
>>>>>       at
>>>>>
>> org.apache.activemq.transport.mqtt.MQTTProtocolConverter.onMQTTCommand(MQTTProtocolConverter.java:204)[activemq-mqtt-5.16.3.jar:5.16.3]
>>>>>       at
>>>>>
>> org.apache.activemq.transport.mqtt.MQTTTransportFilter.onCommand(MQTTTransportFilter.java:94)[activemq-mqtt-5.16.3.jar:5.16.3]
>>>>>       at
>>>>>
>> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)[activemq-client-5.16.3.jar:5.16.3]
>>>>>       at
>>>>>
>> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:233)[activemq-client-5.16.3.jar:5.16.3]
>>>>>       at
>>>>>
>> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215)[activemq-client-5.16.3.jar:5.16.3]
>>>>>       at java.lang.Thread.run(Unknown Source)[:1.8.0_211]
>>>>> 2021-11-02 10:31:19,226 | WARN  | Transport Connection to:
>>>>> tcp://62.83.10.35:3000 failed |
>>>>> org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ
>>>>> Transport: tcp:///62.83.10.35:3000@1883 | localhost            | |
>>>>> org.apache.activemq.transport.mqtt.MQTTProtocolException: Not
>> connected.
>>>>>       at
>>>>>
>> org.apache.activemq.transport.mqtt.MQTTProtocolConverter.checkConnected(MQTTProtocolConverter.java:736)[activemq-mqtt-5.16.3.jar:5.16.3]
>>>>>       at
>>>>>
>> org.apache.activemq.transport.mqtt.MQTTProtocolConverter.onSubscribe(MQTTProtocolConverter.java:377)[activemq-mqtt-5.16.3.jar:5.16.3]
>>>>>       at
>>>>>
>> org.apache.activemq.transport.mqtt.MQTTProtocolConverter.onMQTTCommand(MQTTProtocolConverter.java:212)[activemq-mqtt-5.16.3.jar:5.16.3]
>>>>>       at
>>>>>
>> org.apache.activemq.transport.mqtt.MQTTTransportFilter.onCommand(MQTTTransportFilter.java:94)[activemq-mqtt-5.16.3.jar:5.16.3]
>>>>>       at
>>>>>
>> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)[activemq-client-5.16.3.jar:5.16.3]
>>>>>       at
>>>>>
>> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:233)[activemq-client-5.16.3.jar:5.16.3]
>>>>>       at
>>>>>
>> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215)[activemq-client-5.16.3.jar:5.16.3]
>>>>>       at java.lang.Thread.run(Unknown Source)[:1.8.0_211]
>>>> Regards
>>>>
>>>> Guillaume Cripiau

Re: Connection reset / Already connected / Not connected

Posted by Tim Bain <tb...@alumni.duke.edu>.
What is the CPU load like on the broker host? Is that process spiking the
CPU? If so, can you use JVisualVM or similar to do CPU sampling of the
broker process and find out where it's spending its time?

My suspicion is that the broker is so busy that it's not responding to some
connections fast enough to let them succeed.

Tim

On Wed, Nov 10, 2021, 2:17 AM Guillaume Cripiau <
guillaume.cripiau@murprogroup.com> wrote:

> Hello,
>
> The only other logs (mqtt messages on the server) show this :
> 10:30:45
> remove/ID:airprotec-cta-41267-1635489094200-3:8407/03200646/Connection
> 10:30:46
> add/ID:airprotec-cta-41267-1635489094200-3:8410/03200646/Connection
> 10:30:46
> remove/ID:airprotec-cta-41267-1635489094200-3:8410/03200646/Connection
>
> Yes, the client restarts after 2 hours without connection and the same
> event happens but not always.
>
> The clients try to reconnect, the broker has just restarted.
>
> We cannot reproduce the problem on our dev server with minimal setup.
>
> We actually analyse the TCP connexion : the clients use the source port
> 1883 and the timeout between a CONNECT command and CONNACK is 1 second.
> Can this settings be the cause of our problems?
>
> Regards
>
> Guillaume Cripiau
>
> Le 02-11-21 à 12:45, Tim Bain a écrit :
> > These are broker logs, or client logs? Whichever it is, what's in the
> other
> > process's logs at the same time?
> >
> > You said that reconnecting isn't possible for hours after this happens.
> Do
> > you see the same messages in the logs for that whole time?
> >
> > At the time this happens, what are the client(s) and the broker doing?
> >
> > Can you reproduce the problem reliably in a minimal setup (just one or
> two
> > clients, minimal time and message load)?
> >
> > Tim
> >
> > On Tue, Nov 2, 2021, 5:19 AM Guillaume Cripiau <
> > guillaume.cripiau@murprogroup.com> wrote:
> >
> >> Hello,
> >>
> >> We need support. Many devices disconnect for no apparent reason and
> >> cannot reconnect after several days. Could you help us solve this
> >> problem? Here is an example in the logs :
> >>
> >>> 2021-11-02 10:30:44,413 | WARN  | Transport Connection to:
> >>> tcp://62.83.10.35:1024 failed |
> >>> org.apache.activemq.broker.TransportConnection.Transport | Async
> >>> Exception Handler | localhost            | |
> >>> java.io.IOException: Connection reset
> >>>      at
> >>>
> >>
> org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:46)[activemq-client-5.16.3.jar:5.16.3]
> >>>      at
> >>>
> >>
> org.apache.activemq.transport.mqtt.MQTTTransportFilter.oneway(MQTTTransportFilter.java:83)[activemq-mqtt-5.16.3.jar:5.16.3]
> >>>      at
> >>>
> >>
> org.apache.activemq.transport.mqtt.MQTTInactivityMonitor.doOnewaySend(MQTTInactivityMonitor.java:185)[activemq-mqtt-5.16.3.jar:5.16.3]
> >>>      at
> >>>
> >>
> org.apache.activemq.transport.mqtt.MQTTInactivityMonitor.oneway(MQTTInactivityMonitor.java:174)[activemq-mqtt-5.16.3.jar:5.16.3]
> >>>      at
> >>>
> >>
> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)[activemq-client-5.16.3.jar:5.16.3]
> >>>      at
> >>>
> >>
> org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1485)[activemq-broker-5.16.3.jar:5.16.3]
> >>>      at
> >>>
> >>
> org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:982)[activemq-broker-5.16.3.jar:5.16.3]
> >>>      at
> >>>
> >>
> org.apache.activemq.broker.TransportConnection.iterate(TransportConnection.java:1033)[activemq-broker-5.16.3.jar:5.16.3]
> >>>      at
> >>>
> >>
> org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)[activemq-client-5.16.3.jar:5.16.3]
> >>>      at
> >>>
> >>
> org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)[activemq-client-5.16.3.jar:5.16.3]
> >>>      at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
> >>> Source)[:1.8.0_211]
> >>>      at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> >>> Source)[:1.8.0_211]
> >>>      at java.lang.Thread.run(Unknown Source)[:1.8.0_211]
> >>> 2021-11-02 10:30:46,359 | WARN  | Transport Connection to:
> >>> tcp://62.83.10.35:35598 failed |
> >>> org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ
> >>> Transport: tcp:///62.83.10.35:35598@1883 | localhost            | |
> >>> org.apache.activemq.transport.mqtt.MQTTProtocolException: Already
> >>> connected.
> >>>      at
> >>>
> >>
> org.apache.activemq.transport.mqtt.MQTTProtocolConverter.onMQTTConnect(MQTTProtocolConverter.java:239)[activemq-mqtt-5.16.3.jar:5.16.3]
> >>>      at
> >>>
> >>
> org.apache.activemq.transport.mqtt.MQTTProtocolConverter.onMQTTCommand(MQTTProtocolConverter.java:204)[activemq-mqtt-5.16.3.jar:5.16.3]
> >>>      at
> >>>
> >>
> org.apache.activemq.transport.mqtt.MQTTTransportFilter.onCommand(MQTTTransportFilter.java:94)[activemq-mqtt-5.16.3.jar:5.16.3]
> >>>      at
> >>>
> >>
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)[activemq-client-5.16.3.jar:5.16.3]
> >>>      at
> >>>
> >>
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:233)[activemq-client-5.16.3.jar:5.16.3]
> >>>      at
> >>>
> >>
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215)[activemq-client-5.16.3.jar:5.16.3]
> >>>      at java.lang.Thread.run(Unknown Source)[:1.8.0_211]
> >>> 2021-11-02 10:31:19,226 | WARN  | Transport Connection to:
> >>> tcp://62.83.10.35:3000 failed |
> >>> org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ
> >>> Transport: tcp:///62.83.10.35:3000@1883 | localhost            | |
> >>> org.apache.activemq.transport.mqtt.MQTTProtocolException: Not
> connected.
> >>>      at
> >>>
> >>
> org.apache.activemq.transport.mqtt.MQTTProtocolConverter.checkConnected(MQTTProtocolConverter.java:736)[activemq-mqtt-5.16.3.jar:5.16.3]
> >>>      at
> >>>
> >>
> org.apache.activemq.transport.mqtt.MQTTProtocolConverter.onSubscribe(MQTTProtocolConverter.java:377)[activemq-mqtt-5.16.3.jar:5.16.3]
> >>>      at
> >>>
> >>
> org.apache.activemq.transport.mqtt.MQTTProtocolConverter.onMQTTCommand(MQTTProtocolConverter.java:212)[activemq-mqtt-5.16.3.jar:5.16.3]
> >>>      at
> >>>
> >>
> org.apache.activemq.transport.mqtt.MQTTTransportFilter.onCommand(MQTTTransportFilter.java:94)[activemq-mqtt-5.16.3.jar:5.16.3]
> >>>      at
> >>>
> >>
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)[activemq-client-5.16.3.jar:5.16.3]
> >>>      at
> >>>
> >>
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:233)[activemq-client-5.16.3.jar:5.16.3]
> >>>      at
> >>>
> >>
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215)[activemq-client-5.16.3.jar:5.16.3]
> >>>      at java.lang.Thread.run(Unknown Source)[:1.8.0_211]
> >> Regards
> >>
> >> Guillaume Cripiau
>

Re: Connection reset / Already connected / Not connected

Posted by Guillaume Cripiau <gu...@murprogroup.com>.
Hello,

The only other logs (mqtt messages on the server) show this :
10:30:45 
remove/ID:airprotec-cta-41267-1635489094200-3:8407/03200646/Connection
10:30:46 add/ID:airprotec-cta-41267-1635489094200-3:8410/03200646/Connection
10:30:46 
remove/ID:airprotec-cta-41267-1635489094200-3:8410/03200646/Connection

Yes, the client restarts after 2 hours without connection and the same 
event happens but not always.

The clients try to reconnect, the broker has just restarted.

We cannot reproduce the problem on our dev server with minimal setup.

We actually analyse the TCP connexion : the clients use the source port 
1883 and the timeout between a CONNECT command and CONNACK is 1 second. 
Can this settings be the cause of our problems?

Regards

Guillaume Cripiau

Le 02-11-21 à 12:45, Tim Bain a écrit :
> These are broker logs, or client logs? Whichever it is, what's in the other
> process's logs at the same time?
>
> You said that reconnecting isn't possible for hours after this happens. Do
> you see the same messages in the logs for that whole time?
>
> At the time this happens, what are the client(s) and the broker doing?
>
> Can you reproduce the problem reliably in a minimal setup (just one or two
> clients, minimal time and message load)?
>
> Tim
>
> On Tue, Nov 2, 2021, 5:19 AM Guillaume Cripiau <
> guillaume.cripiau@murprogroup.com> wrote:
>
>> Hello,
>>
>> We need support. Many devices disconnect for no apparent reason and
>> cannot reconnect after several days. Could you help us solve this
>> problem? Here is an example in the logs :
>>
>>> 2021-11-02 10:30:44,413 | WARN  | Transport Connection to:
>>> tcp://62.83.10.35:1024 failed |
>>> org.apache.activemq.broker.TransportConnection.Transport | Async
>>> Exception Handler | localhost            | |
>>> java.io.IOException: Connection reset
>>>      at
>>>
>> org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:46)[activemq-client-5.16.3.jar:5.16.3]
>>>      at
>>>
>> org.apache.activemq.transport.mqtt.MQTTTransportFilter.oneway(MQTTTransportFilter.java:83)[activemq-mqtt-5.16.3.jar:5.16.3]
>>>      at
>>>
>> org.apache.activemq.transport.mqtt.MQTTInactivityMonitor.doOnewaySend(MQTTInactivityMonitor.java:185)[activemq-mqtt-5.16.3.jar:5.16.3]
>>>      at
>>>
>> org.apache.activemq.transport.mqtt.MQTTInactivityMonitor.oneway(MQTTInactivityMonitor.java:174)[activemq-mqtt-5.16.3.jar:5.16.3]
>>>      at
>>>
>> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)[activemq-client-5.16.3.jar:5.16.3]
>>>      at
>>>
>> org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1485)[activemq-broker-5.16.3.jar:5.16.3]
>>>      at
>>>
>> org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:982)[activemq-broker-5.16.3.jar:5.16.3]
>>>      at
>>>
>> org.apache.activemq.broker.TransportConnection.iterate(TransportConnection.java:1033)[activemq-broker-5.16.3.jar:5.16.3]
>>>      at
>>>
>> org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)[activemq-client-5.16.3.jar:5.16.3]
>>>      at
>>>
>> org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)[activemq-client-5.16.3.jar:5.16.3]
>>>      at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
>>> Source)[:1.8.0_211]
>>>      at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
>>> Source)[:1.8.0_211]
>>>      at java.lang.Thread.run(Unknown Source)[:1.8.0_211]
>>> 2021-11-02 10:30:46,359 | WARN  | Transport Connection to:
>>> tcp://62.83.10.35:35598 failed |
>>> org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ
>>> Transport: tcp:///62.83.10.35:35598@1883 | localhost            | |
>>> org.apache.activemq.transport.mqtt.MQTTProtocolException: Already
>>> connected.
>>>      at
>>>
>> org.apache.activemq.transport.mqtt.MQTTProtocolConverter.onMQTTConnect(MQTTProtocolConverter.java:239)[activemq-mqtt-5.16.3.jar:5.16.3]
>>>      at
>>>
>> org.apache.activemq.transport.mqtt.MQTTProtocolConverter.onMQTTCommand(MQTTProtocolConverter.java:204)[activemq-mqtt-5.16.3.jar:5.16.3]
>>>      at
>>>
>> org.apache.activemq.transport.mqtt.MQTTTransportFilter.onCommand(MQTTTransportFilter.java:94)[activemq-mqtt-5.16.3.jar:5.16.3]
>>>      at
>>>
>> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)[activemq-client-5.16.3.jar:5.16.3]
>>>      at
>>>
>> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:233)[activemq-client-5.16.3.jar:5.16.3]
>>>      at
>>>
>> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215)[activemq-client-5.16.3.jar:5.16.3]
>>>      at java.lang.Thread.run(Unknown Source)[:1.8.0_211]
>>> 2021-11-02 10:31:19,226 | WARN  | Transport Connection to:
>>> tcp://62.83.10.35:3000 failed |
>>> org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ
>>> Transport: tcp:///62.83.10.35:3000@1883 | localhost            | |
>>> org.apache.activemq.transport.mqtt.MQTTProtocolException: Not connected.
>>>      at
>>>
>> org.apache.activemq.transport.mqtt.MQTTProtocolConverter.checkConnected(MQTTProtocolConverter.java:736)[activemq-mqtt-5.16.3.jar:5.16.3]
>>>      at
>>>
>> org.apache.activemq.transport.mqtt.MQTTProtocolConverter.onSubscribe(MQTTProtocolConverter.java:377)[activemq-mqtt-5.16.3.jar:5.16.3]
>>>      at
>>>
>> org.apache.activemq.transport.mqtt.MQTTProtocolConverter.onMQTTCommand(MQTTProtocolConverter.java:212)[activemq-mqtt-5.16.3.jar:5.16.3]
>>>      at
>>>
>> org.apache.activemq.transport.mqtt.MQTTTransportFilter.onCommand(MQTTTransportFilter.java:94)[activemq-mqtt-5.16.3.jar:5.16.3]
>>>      at
>>>
>> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)[activemq-client-5.16.3.jar:5.16.3]
>>>      at
>>>
>> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:233)[activemq-client-5.16.3.jar:5.16.3]
>>>      at
>>>
>> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215)[activemq-client-5.16.3.jar:5.16.3]
>>>      at java.lang.Thread.run(Unknown Source)[:1.8.0_211]
>> Regards
>>
>> Guillaume Cripiau

Re: Connection reset / Already connected / Not connected

Posted by Tim Bain <tb...@alumni.duke.edu>.
These are broker logs, or client logs? Whichever it is, what's in the other
process's logs at the same time?

You said that reconnecting isn't possible for hours after this happens. Do
you see the same messages in the logs for that whole time?

At the time this happens, what are the client(s) and the broker doing?

Can you reproduce the problem reliably in a minimal setup (just one or two
clients, minimal time and message load)?

Tim

On Tue, Nov 2, 2021, 5:19 AM Guillaume Cripiau <
guillaume.cripiau@murprogroup.com> wrote:

> Hello,
>
> We need support. Many devices disconnect for no apparent reason and
> cannot reconnect after several days. Could you help us solve this
> problem? Here is an example in the logs :
>
> > 2021-11-02 10:30:44,413 | WARN  | Transport Connection to:
> > tcp://62.83.10.35:1024 failed |
> > org.apache.activemq.broker.TransportConnection.Transport | Async
> > Exception Handler | localhost            | |
> > java.io.IOException: Connection reset
> >     at
> >
> org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:46)[activemq-client-5.16.3.jar:5.16.3]
> >     at
> >
> org.apache.activemq.transport.mqtt.MQTTTransportFilter.oneway(MQTTTransportFilter.java:83)[activemq-mqtt-5.16.3.jar:5.16.3]
> >     at
> >
> org.apache.activemq.transport.mqtt.MQTTInactivityMonitor.doOnewaySend(MQTTInactivityMonitor.java:185)[activemq-mqtt-5.16.3.jar:5.16.3]
> >     at
> >
> org.apache.activemq.transport.mqtt.MQTTInactivityMonitor.oneway(MQTTInactivityMonitor.java:174)[activemq-mqtt-5.16.3.jar:5.16.3]
> >     at
> >
> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)[activemq-client-5.16.3.jar:5.16.3]
> >     at
> >
> org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1485)[activemq-broker-5.16.3.jar:5.16.3]
> >     at
> >
> org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:982)[activemq-broker-5.16.3.jar:5.16.3]
> >     at
> >
> org.apache.activemq.broker.TransportConnection.iterate(TransportConnection.java:1033)[activemq-broker-5.16.3.jar:5.16.3]
> >     at
> >
> org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)[activemq-client-5.16.3.jar:5.16.3]
> >     at
> >
> org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)[activemq-client-5.16.3.jar:5.16.3]
> >     at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
> > Source)[:1.8.0_211]
> >     at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> > Source)[:1.8.0_211]
> >     at java.lang.Thread.run(Unknown Source)[:1.8.0_211]
> > 2021-11-02 10:30:46,359 | WARN  | Transport Connection to:
> > tcp://62.83.10.35:35598 failed |
> > org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ
> > Transport: tcp:///62.83.10.35:35598@1883 | localhost            | |
> > org.apache.activemq.transport.mqtt.MQTTProtocolException: Already
> > connected.
> >     at
> >
> org.apache.activemq.transport.mqtt.MQTTProtocolConverter.onMQTTConnect(MQTTProtocolConverter.java:239)[activemq-mqtt-5.16.3.jar:5.16.3]
> >     at
> >
> org.apache.activemq.transport.mqtt.MQTTProtocolConverter.onMQTTCommand(MQTTProtocolConverter.java:204)[activemq-mqtt-5.16.3.jar:5.16.3]
> >     at
> >
> org.apache.activemq.transport.mqtt.MQTTTransportFilter.onCommand(MQTTTransportFilter.java:94)[activemq-mqtt-5.16.3.jar:5.16.3]
> >     at
> >
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)[activemq-client-5.16.3.jar:5.16.3]
> >     at
> >
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:233)[activemq-client-5.16.3.jar:5.16.3]
> >     at
> >
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215)[activemq-client-5.16.3.jar:5.16.3]
> >     at java.lang.Thread.run(Unknown Source)[:1.8.0_211]
> > 2021-11-02 10:31:19,226 | WARN  | Transport Connection to:
> > tcp://62.83.10.35:3000 failed |
> > org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ
> > Transport: tcp:///62.83.10.35:3000@1883 | localhost            | |
> > org.apache.activemq.transport.mqtt.MQTTProtocolException: Not connected.
> >     at
> >
> org.apache.activemq.transport.mqtt.MQTTProtocolConverter.checkConnected(MQTTProtocolConverter.java:736)[activemq-mqtt-5.16.3.jar:5.16.3]
> >     at
> >
> org.apache.activemq.transport.mqtt.MQTTProtocolConverter.onSubscribe(MQTTProtocolConverter.java:377)[activemq-mqtt-5.16.3.jar:5.16.3]
> >     at
> >
> org.apache.activemq.transport.mqtt.MQTTProtocolConverter.onMQTTCommand(MQTTProtocolConverter.java:212)[activemq-mqtt-5.16.3.jar:5.16.3]
> >     at
> >
> org.apache.activemq.transport.mqtt.MQTTTransportFilter.onCommand(MQTTTransportFilter.java:94)[activemq-mqtt-5.16.3.jar:5.16.3]
> >     at
> >
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)[activemq-client-5.16.3.jar:5.16.3]
> >     at
> >
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:233)[activemq-client-5.16.3.jar:5.16.3]
> >     at
> >
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215)[activemq-client-5.16.3.jar:5.16.3]
> >     at java.lang.Thread.run(Unknown Source)[:1.8.0_211]
>
> Regards
>
> Guillaume Cripiau