You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@activemq.apache.org by "Xinjing Wei (Jira)" <ji...@apache.org> on 2021/06/24 23:05:00 UTC
[jira] [Updated] (AMQ-8306) Race Condition at ActiveMQ startup when
using MQTT connections
[ https://issues.apache.org/jira/browse/AMQ-8306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Xinjing Wei updated AMQ-8306:
-----------------------------
Description:
This is AWS Amazon MQ team.
We have found a race condition during Active MQ start up that would cause the broker to fail the startup process.
The race condition happens when using MQTT to connect to the broker before the broker is fully started. In the Active MQ code, the transport connectors are the first to start, right after the MQTT transport connector is started (but not all services are started), if an MQTT connection is received by the broker, the org.apache.activemq.transport.mqtt.MQTTPacketIdGenerator service will be added to the list of broker services to start (code references: [MQTTPacketIdGenerator|https://github.com/apache/activemq/blob/activemq-5.15.15/activemq-mqtt/src/main/java/org/apache/activemq/transport/mqtt/MQTTPacketIdGenerator.java#L107], [BrokerService|https://github.com/apache/activemq/blob/activemq-5.15.15/activemq-broker/src/main/java/org/apache/activemq/broker/BrokerService.java#L2699]), causing a modification on the Service list while the list is iterated. The broker startup will fail with the following exception:
{noformat}
2021-06-24 15:33:11,436 | ERROR | Failed to start Apache ActiveMQ (localhost, ID:8c85904c8822-51824-1624573980902-0:1) | org.apache.activemq.broker.BrokerService | main
java.util.ConcurrentModificationException
at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:909)[:1.8.0_231]
at java.util.ArrayList$Itr.next(ArrayList.java:859)[:1.8.0_231]
at org.apache.activemq.broker.BrokerService.startAllConnectors(BrokerService.java:2698)[activemq-broker-5.15.15.jar:5.15.15]
at org.apache.activemq.broker.BrokerService.doStartBroker(BrokerService.java:777)[activemq-broker-5.15.15.jar:5.15.15]
at org.apache.activemq.broker.BrokerService.startBroker(BrokerService.java:739)[activemq-broker-5.15.15.jar:5.15.15]
at org.apache.activemq.broker.BrokerService.start(BrokerService.java:642)[activemq-broker-5.15.15.jar:5.15.15]
at org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:73)[activemq-spring-5.15.15.jar:5.15.15]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:1.8.0_231]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[:1.8.0_231]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.8.0_231]
at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_231]{noformat}
Here are the steps to reproduce:
# Add a sleep at [https://github.com/apache/activemq/blob/activemq-5.15.15/activemq-broker/src/main/java/org/apache/activemq/broker/BrokerService.java#L2697] so that we can guarantee the race condition to happen
# Add a log in [https://github.com/apache/activemq/blob/activemq-5.15.15/activemq-broker/src/main/java/org/apache/activemq/broker/BrokerService.java#L1484] to print out the services being added
# build and start the ActiveMQ process
# create a producer to send some transacted messages through MQTT:
{code:java}
./bin/activemq producer --brokerUrl failover://mqtt://localhost:1883 --user admin --password administrator --messageSize 1000 --messageCount 100000 --destination TEST
{code}
# before the above command can finish, stop the ActiveMQ process
# restart the ActiveMQ process
# As soon as you see "Connector mqtt started", execute the command in step 3 again.
Here I attach the activemq.log being produced by my tests: [^activemq.log]
was:
This is AWS Amazon MQ team.
We have found a race condition during Active MQ start up that would cause the broker to fail the startup process.
The race condition happens when using MQTT to connect to the broker before the broker is fully started. In the Active MQ code, the transport connectors are the first to start, right after the MQTT transport connector is started (but not all services are started), if an MQTT connection is received by the broker, the org.apache.activemq.transport.mqtt.MQTTPacketIdGenerator service will be added to the list of broker services to start (code references: [MQTTPacketIdGenerator|https://github.com/apache/activemq/blob/activemq-5.15.15/activemq-mqtt/src/main/java/org/apache/activemq/transport/mqtt/MQTTPacketIdGenerator.java#L107], [BrokerService|https://github.com/apache/activemq/blob/activemq-5.15.15/activemq-broker/src/main/java/org/apache/activemq/broker/BrokerService.java#L2699]), causing a modification on the Service list while the list is iterated. The broker startup will fail with the following exception:
{noformat}
2021-06-24 15:33:11,436 | ERROR | Failed to start Apache ActiveMQ (localhost, ID:8c85904c8822-51824-1624573980902-0:1) | org.apache.activemq.broker.BrokerService | main
java.util.ConcurrentModificationException
at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:909)[:1.8.0_231]
at java.util.ArrayList$Itr.next(ArrayList.java:859)[:1.8.0_231]
at org.apache.activemq.broker.BrokerService.startAllConnectors(BrokerService.java:2698)[activemq-broker-5.15.15.jar:5.15.15]
at org.apache.activemq.broker.BrokerService.doStartBroker(BrokerService.java:777)[activemq-broker-5.15.15.jar:5.15.15]
at org.apache.activemq.broker.BrokerService.startBroker(BrokerService.java:739)[activemq-broker-5.15.15.jar:5.15.15]
at org.apache.activemq.broker.BrokerService.start(BrokerService.java:642)[activemq-broker-5.15.15.jar:5.15.15]
at org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:73)[activemq-spring-5.15.15.jar:5.15.15]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:1.8.0_231]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[:1.8.0_231]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.8.0_231]
at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_231]{noformat}
Here are the steps to reproduce:
# Add a sleep at [https://github.com/apache/activemq/blob/activemq-5.15.15/activemq-broker/src/main/java/org/apache/activemq/broker/BrokerService.java#L2697]
# Add a log in [https://github.com/apache/activemq/blob/activemq-5.15.15/activemq-broker/src/main/java/org/apache/activemq/broker/BrokerService.java#L1484] to print out the services being added
# build and start the ActiveMQ process
# create a producer to send some transacted messages through MQTT:
{code:java}
./bin/activemq producer --brokerUrl failover://mqtt://localhost:1883 --user admin --password administrator --messageSize 1000 --messageCount 100000 --destination TEST
{code}
# before the above command can finish, stop the ActiveMQ process
# restart the ActiveMQ process
# As soon as you see "Connector mqtt started", execute the command in step 3 again.
Here I attach the activemq.log being produced by my tests: [^activemq.log]
> Race Condition at ActiveMQ startup when using MQTT connections
> --------------------------------------------------------------
>
> Key: AMQ-8306
> URL: https://issues.apache.org/jira/browse/AMQ-8306
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.15.15
> Reporter: Xinjing Wei
> Priority: Major
> Attachments: activemq.log
>
>
> This is AWS Amazon MQ team.
>
> We have found a race condition during Active MQ start up that would cause the broker to fail the startup process.
>
> The race condition happens when using MQTT to connect to the broker before the broker is fully started. In the Active MQ code, the transport connectors are the first to start, right after the MQTT transport connector is started (but not all services are started), if an MQTT connection is received by the broker, the org.apache.activemq.transport.mqtt.MQTTPacketIdGenerator service will be added to the list of broker services to start (code references: [MQTTPacketIdGenerator|https://github.com/apache/activemq/blob/activemq-5.15.15/activemq-mqtt/src/main/java/org/apache/activemq/transport/mqtt/MQTTPacketIdGenerator.java#L107], [BrokerService|https://github.com/apache/activemq/blob/activemq-5.15.15/activemq-broker/src/main/java/org/apache/activemq/broker/BrokerService.java#L2699]), causing a modification on the Service list while the list is iterated. The broker startup will fail with the following exception:
> {noformat}
> 2021-06-24 15:33:11,436 | ERROR | Failed to start Apache ActiveMQ (localhost, ID:8c85904c8822-51824-1624573980902-0:1) | org.apache.activemq.broker.BrokerService | main
> java.util.ConcurrentModificationException
> at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:909)[:1.8.0_231]
> at java.util.ArrayList$Itr.next(ArrayList.java:859)[:1.8.0_231]
> at org.apache.activemq.broker.BrokerService.startAllConnectors(BrokerService.java:2698)[activemq-broker-5.15.15.jar:5.15.15]
> at org.apache.activemq.broker.BrokerService.doStartBroker(BrokerService.java:777)[activemq-broker-5.15.15.jar:5.15.15]
> at org.apache.activemq.broker.BrokerService.startBroker(BrokerService.java:739)[activemq-broker-5.15.15.jar:5.15.15]
> at org.apache.activemq.broker.BrokerService.start(BrokerService.java:642)[activemq-broker-5.15.15.jar:5.15.15]
> at org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:73)[activemq-spring-5.15.15.jar:5.15.15]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:1.8.0_231]
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[:1.8.0_231]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.8.0_231]
> at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_231]{noformat}
>
> Here are the steps to reproduce:
> # Add a sleep at [https://github.com/apache/activemq/blob/activemq-5.15.15/activemq-broker/src/main/java/org/apache/activemq/broker/BrokerService.java#L2697] so that we can guarantee the race condition to happen
> # Add a log in [https://github.com/apache/activemq/blob/activemq-5.15.15/activemq-broker/src/main/java/org/apache/activemq/broker/BrokerService.java#L1484] to print out the services being added
> # build and start the ActiveMQ process
> # create a producer to send some transacted messages through MQTT:
> {code:java}
> ./bin/activemq producer --brokerUrl failover://mqtt://localhost:1883 --user admin --password administrator --messageSize 1000 --messageCount 100000 --destination TEST
> {code}
> # before the above command can finish, stop the ActiveMQ process
> # restart the ActiveMQ process
> # As soon as you see "Connector mqtt started", execute the command in step 3 again.
>
> Here I attach the activemq.log being produced by my tests: [^activemq.log]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)