You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@activemq.apache.org by "Stirling Chow (JIRA)" <ji...@apache.org> on 2010/11/03 23:57:00 UTC
[jira] Created: (AMQ-3014) DemandForwardingBridgeSupport can send
BrokerInfo to remote transport before local broker ID is known.
DemandForwardingBridgeSupport can send BrokerInfo to remote transport before local broker ID is known.
------------------------------------------------------------------------------------------------------
Key: AMQ-3014
URL: https://issues.apache.org/activemq/browse/AMQ-3014
Project: ActiveMQ
Issue Type: Bug
Components: Broker, Transport
Affects Versions: 5.4.1
Reporter: Stirling Chow
Symptom
========
We have a production system that involves a set of Brokers connected in a demand-forwarding Network-of-Brokers using HTTP-based bridges. Each Broker periodically scans its list of peer brokers by iterating over RegionBroker.getPeerBrokerInfos:
public synchronized BrokerInfo[] getPeerBrokerInfos() {
BrokerInfo[] result = new BrokerInfo[brokerInfos.size()];
result = brokerInfos.toArray(result);
return result;
}
This scanning code assumes that BrokerInfo.getBrokerId() is always non-null (since every broker should have an ID). However, we periodically noticed that BrokerInfo.getBrokerId() returned a NULL value, which was very unexpected.
Cause
======
We analyzed the DemandForwardingBridgeSupport and noticed that when the remote bridge/transport is started, it sends the local Broker's ID:
protected void startRemoteBridge() throws Exception {
...
brokerInfo.setBrokerId(this.localBrokerId);
remoteBroker.oneway(brokerInfo);
}
The local Broker's ID is not initially known until it is received from the local transport and processed by DemandForwardingBridge.serviceLocalBrokerInfo(...):
protected void serviceLocalBrokerInfo(Command command) throws InterruptedException {
synchronized (brokerInfoMutex) {
localBrokerId = ((BrokerInfo)command).getBrokerId();
localBrokerPath[0] = localBrokerId;
localBrokerIdKnownLatch.countDown();
The local Broker's ID is dispatched asynchronously when the local transport is started, as seen in TransportConnection.start():
public void start() throws Exception {
starting = true;
try {
synchronized (this) {
if (taskRunnerFactory != null) {
taskRunner = taskRunnerFactory.createTaskRunner(this, "ActiveMQ Connection Dispatcher: "
+ getRemoteAddress());
} else {
taskRunner = null;
}
transport.start();
active = true;
dispatchAsync(connector.getBrokerInfo());
Because of the asynchronous dispatch, the remote bridge may be started before the local Broker's ID is known. This would be particularly evident when the local broker is under load processing a lot of tasks.
We've attached a unit test that demonstrates how a slow asynchronous dispatch on the local transport can cause the remote transport to transmit a null BrokerId.
Solution
======
DemandForwardingBridgeSupport already contains a localBrokerIdKnownLatch, so starting the remote transport should wait for this latch before accessing the local Broker's ID (see patch).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AMQ-3014) DemandForwardingBridgeSupport can send
BrokerInfo to remote transport before local broker ID is known.
Posted by "Stirling Chow (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-3014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Stirling Chow updated AMQ-3014:
-------------------------------
Attachment: patch.txt
Patch of DemandForwardingBridgeSupport, which waits until the local Broker's ID is known before transmitting the BrokerInfo to the remote transport.
> DemandForwardingBridgeSupport can send BrokerInfo to remote transport before local broker ID is known.
> ------------------------------------------------------------------------------------------------------
>
> Key: AMQ-3014
> URL: https://issues.apache.org/activemq/browse/AMQ-3014
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker, Transport
> Affects Versions: 5.4.1
> Reporter: Stirling Chow
> Attachments: patch.txt
>
>
> Symptom
> ========
> We have a production system that involves a set of Brokers connected in a demand-forwarding Network-of-Brokers using HTTP-based bridges. Each Broker periodically scans its list of peer brokers by iterating over RegionBroker.getPeerBrokerInfos:
> public synchronized BrokerInfo[] getPeerBrokerInfos() {
> BrokerInfo[] result = new BrokerInfo[brokerInfos.size()];
> result = brokerInfos.toArray(result);
> return result;
> }
> This scanning code assumes that BrokerInfo.getBrokerId() is always non-null (since every broker should have an ID). However, we periodically noticed that BrokerInfo.getBrokerId() returned a NULL value, which was very unexpected.
> Cause
> ======
> We analyzed the DemandForwardingBridgeSupport and noticed that when the remote bridge/transport is started, it sends the local Broker's ID:
> protected void startRemoteBridge() throws Exception {
> ...
> brokerInfo.setBrokerId(this.localBrokerId);
> remoteBroker.oneway(brokerInfo);
> }
> The local Broker's ID is not initially known until it is received from the local transport and processed by DemandForwardingBridge.serviceLocalBrokerInfo(...):
> protected void serviceLocalBrokerInfo(Command command) throws InterruptedException {
> synchronized (brokerInfoMutex) {
> localBrokerId = ((BrokerInfo)command).getBrokerId();
> localBrokerPath[0] = localBrokerId;
> localBrokerIdKnownLatch.countDown();
> The local Broker's ID is dispatched asynchronously when the local transport is started, as seen in TransportConnection.start():
> public void start() throws Exception {
> starting = true;
> try {
> synchronized (this) {
> if (taskRunnerFactory != null) {
> taskRunner = taskRunnerFactory.createTaskRunner(this, "ActiveMQ Connection Dispatcher: "
> + getRemoteAddress());
> } else {
> taskRunner = null;
> }
> transport.start();
> active = true;
> dispatchAsync(connector.getBrokerInfo());
> Because of the asynchronous dispatch, the remote bridge may be started before the local Broker's ID is known. This would be particularly evident when the local broker is under load processing a lot of tasks.
> We've attached a unit test that demonstrates how a slow asynchronous dispatch on the local transport can cause the remote transport to transmit a null BrokerId.
> Solution
> ======
> DemandForwardingBridgeSupport already contains a localBrokerIdKnownLatch, so starting the remote transport should wait for this latch before accessing the local Broker's ID (see patch).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AMQ-3014) DemandForwardingBridgeSupport can send
BrokerInfo to remote transport before local broker ID is known.
Posted by "Stirling Chow (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-3014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Stirling Chow updated AMQ-3014:
-------------------------------
Attachment: NullBrokerIdTest.java
Unit test that demonstrates the ability of DemandForwardingBridgeSupport to transmit a null local Broker ID. The test contains a "normal" case that typically succeeds because there are few delays in VMTransport. The test also contains a "delayed" case that fails with the unpatched DemandForwardingBridgeSupport. When the attached patch.txt is applied, both tests pass.
> DemandForwardingBridgeSupport can send BrokerInfo to remote transport before local broker ID is known.
> ------------------------------------------------------------------------------------------------------
>
> Key: AMQ-3014
> URL: https://issues.apache.org/activemq/browse/AMQ-3014
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker, Transport
> Affects Versions: 5.4.1
> Reporter: Stirling Chow
> Attachments: NullBrokerIdTest.java, patch.txt
>
>
> Symptom
> ========
> We have a production system that involves a set of Brokers connected in a demand-forwarding Network-of-Brokers using HTTP-based bridges. Each Broker periodically scans its list of peer brokers by iterating over RegionBroker.getPeerBrokerInfos:
> public synchronized BrokerInfo[] getPeerBrokerInfos() {
> BrokerInfo[] result = new BrokerInfo[brokerInfos.size()];
> result = brokerInfos.toArray(result);
> return result;
> }
> This scanning code assumes that BrokerInfo.getBrokerId() is always non-null (since every broker should have an ID). However, we periodically noticed that BrokerInfo.getBrokerId() returned a NULL value, which was very unexpected.
> Cause
> ======
> We analyzed the DemandForwardingBridgeSupport and noticed that when the remote bridge/transport is started, it sends the local Broker's ID:
> protected void startRemoteBridge() throws Exception {
> ...
> brokerInfo.setBrokerId(this.localBrokerId);
> remoteBroker.oneway(brokerInfo);
> }
> The local Broker's ID is not initially known until it is received from the local transport and processed by DemandForwardingBridge.serviceLocalBrokerInfo(...):
> protected void serviceLocalBrokerInfo(Command command) throws InterruptedException {
> synchronized (brokerInfoMutex) {
> localBrokerId = ((BrokerInfo)command).getBrokerId();
> localBrokerPath[0] = localBrokerId;
> localBrokerIdKnownLatch.countDown();
> The local Broker's ID is dispatched asynchronously when the local transport is started, as seen in TransportConnection.start():
> public void start() throws Exception {
> starting = true;
> try {
> synchronized (this) {
> if (taskRunnerFactory != null) {
> taskRunner = taskRunnerFactory.createTaskRunner(this, "ActiveMQ Connection Dispatcher: "
> + getRemoteAddress());
> } else {
> taskRunner = null;
> }
> transport.start();
> active = true;
> dispatchAsync(connector.getBrokerInfo());
> Because of the asynchronous dispatch, the remote bridge may be started before the local Broker's ID is known. This would be particularly evident when the local broker is under load processing a lot of tasks.
> We've attached a unit test that demonstrates how a slow asynchronous dispatch on the local transport can cause the remote transport to transmit a null BrokerId.
> Solution
> ======
> DemandForwardingBridgeSupport already contains a localBrokerIdKnownLatch, so starting the remote transport should wait for this latch before accessing the local Broker's ID (see patch).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.