[jira] [Created] (AMQ-6316) Stuck messages to due of duplicated NetworkBrokerId?

["Johan Carlquist (JIRA)" <ji...@apache.org>]

Johan Carlquist created AMQ-6316:
------------------------------------

             Summary: Stuck messages to due of duplicated NetworkBrokerId?
                 Key: AMQ-6316
                 URL: https://issues.apache.org/jira/browse/AMQ-6316
             Project: ActiveMQ
          Issue Type: Bug
          Components: Broker
    Affects Versions: 5.13.0
         Environment: # lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 14.04.4 LTS
Release:        14.04
Codename:       trusty

# /local/jdk/bin/java -version
java version "1.8.0_45"
Java(TM) SE Runtime Environment (build 1.8.0_45-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode)
            Reporter: Johan Carlquist
            Priority: Critical


We have trouble with stuck messages in our network of brokers.
Our setup is two brokers and a loadbalancer in front of them:
{code}
           ┌───────┐
           │ F5 LB │
           └───────┘
               │
       ┌───────┴───────┐
       ▼               ▼
┌─────────────┐ ┌─────────────┐
│esb-test-mq01│ │esb-test-mq02│
└─────────────┘ └─────────────┘
{code}

Our relevant\(?) config:
{code}
[...]
<networkConnectors>
  <networkConnector
    networkTTL="-1"
    conduitSubscriptions="false"
    decreaseNetworkConsumerPriority="true"
    uri="static:(ssl://${activemq.networkPartner}:61616)"
    userName="${activemq.username}"
    password="${activemq.password}"
  />
</networkConnectors>
[...]
<destinationPolicy>
  <policyMap>
    <policyEntries>
      <policyEntry queue=">">
        <networkBridgeFilterFactory>
          <conditionalNetworkBridgeFilterFactory replayWhenNoConsumers="true"/>
        </networkBridgeFilterFactory>
      </policyEntry>
    </policyEntries>
  </policyMap>
</destinationPolicy>
[...]
{code}

The trace log says:
{code}
{
    "endOfBatch": false,
    "level": "TRACE",
    "loggerFqcn": "org.slf4j.impl.Log4jLoggerAdapter",
    "loggerName": "org.apache.activemq.command.NetworkBridgeFilter",
    "message": "Message all ready routed once through target broker (ID:esb-test-mq02-56500-1465365652843-1:1), path: [ID:esb-test-mq02-35507-1463568719599-0:1, ID:esb-test-mq01-57539-1464335954620-0:1, ID:esb-test-mq02-34785-1464595152761-0:1, ID:esb-test-mq01-57539-1464335954620-0:1, ID:esb-test-mq02-34785-1464595152761-0:1, ID:esb-test-mq01-57539-1464335954620-0:1, ID:esb-test-mq02-41348-1464783144640-0:1, ID:esb-test-mq01-51394-1465301962265-1:1, ID:esb-test-mq02-56500-1465365652843-1:1] - ignoring: ActiveMQTextMessage {commandId = 28628, responseRequired = true, messageId = ID:esb-test-app02-46742-1461830778786-1:183:1:1:150, originalDestination = null, originalTransactionId = null, producerId = esb-test-mq02.it.su.se->esb-test-mq01.it.su.se-56500-1465365652843-6:2:1:1, destination = queue://su.it.sukat.users.polopoly, transactionId = null, expiration = 0, timestamp = 1463591178675, arrival = 0, brokerInTime = 1465367541509, brokerOutTime = 1465367541509, correlationId = null, replyTo = null, persistent = true, type = null, priority = 4, groupID = null, groupSequence = 0, targetConsumerId = null, compressed = false, userID = null, content = org.apache.activemq.util.ByteSequence@1d8fbb40, marshalledProperties = org.apache.activemq.util.ByteSequence@58407d25, dataStructure = null, redeliveryCounter = 0, size = 2711, properties = {payload_version=1, CamelHttpCharacterEncoding=ISO-8859-1, operationName=defaultResourceMethod, X_HYPHEN_Forwarded_HYPHEN_For=77.238.37.118, user_HYPHEN_agent=libwww-perl/6.05, Content_HYPHEN_Length=312, TE=deflate,gzip;q=0.3, breadcrumbId=ID-esb-test-app02-45469-1461830360513-4-49863, Content_HYPHEN_Type=application/x-www-form-urlencoded, CamelHttpMethod=POST, host=esb-test.it.su.se, CamelAcceptContentType=*/*, connection=TE, close, CamelHttpUri=/cxf/sukat/, CamelHttpPath=/}, readOnlyProperties = false, readOnlyBody = false, droppable = false, jmsXGroupFirstForConsumer = false, text = "text",
    "thread": "ActiveMQ BrokerService[esb-test-mq01.it.su.se] Task-14",
    "timeMillis": 1465370023382
}
{code}

As you can see from the log the message has jumped between the two brokers "esb-test-mq01" or "esb-test-mq02" several times, which is fine and what we want it to do.
But then the jumping stops. And from what we can determine from the log is that the NetworkBrokerId is used twice and the following code rejects jumping of the message:
https://github.com/apache/activemq/blob/cc81680/activemq-client/src/main/java/org/apache/activemq/command/NetworkBridgeFilter.java#L85-L92

Have we missed something obvious here? 

Is this a bug or is it working as intended? How can we work around it?




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)