broker network disconnected 30mins after producer's tempUsage full

["yang.yang.zz" <ya...@outlook.com>]

Hi:

I have a very simple broker network. Broker-1 and Broker-2. There's is a
Producer application and Consumer application. The network looks like:

Producer <--> Broker-1    <----- duplex network connector -----> Broker-2
<---> Consumer

The problem is, if Consumer is slow or down. The produced data will
accumulate at Broker-1's tempUsage. Eventually, when the tempUsage is full.
Then if I start Consumer right away. It will still be able to collect all
the pending message from Broker-1. But if I start Consumer after 30 mins.
The consumer will never collect any pending message from Broker-1. It just
seems the network connection from Broker-1 and Broker-2 is permanently
broken.

At the around the 30mins when tempUsage is full, there're error messages
seen in Broker-1 and Broker-2

Broker-1

2015-07-27 00:35:25,207 | ERROR | Exception:
org.apache.activemq.transport.InactivityIOException: Cannot send, channel
has already failed: tcp://138.42.104.75:61616 on duplex forward of:
ActiveMQTextMessage {commandId = 63, responseRequired = true, messageId =
ID:lager.ca.com-41963-1437971724789-1:2:61:1:1, originalDestination = null,
originalTransactionId = null, producerId =
dc_broker_dd2413a4-c930-4082-be05-426319724f88->da_broker-47643-1437954343579-209:2:1:1,
destination =
queue://DIP-poll.responses.irep-polperfdc01.ca.com:dd2413a4-c930-4082-be05-426319724f88,
transactionId = null, expiration = 0, timestamp = 1437971725412, arrival =
0, brokerInTime = 1437974529971, brokerOutTime = 1437974529956,
correlationId = 7f19ac51-274e-4b3a-a76e-3c727513384e, replyTo = null,
persistent = false, type = java.lang.Throwable, priority = 4, groupID =
null, groupSequence = 0, targetConsumerId = null, compressed = true, userID
= null, content = org.apache.activemq.util.ByteSequence@30af7675,
marshalledProperties = null, dataStructure = null, redeliveryCounter = 0,
size = 0, properties = null, readOnlyProperties = false, readOnlyBody =
false, droppable = false, jmsXGroupFirstForConsumer = false, text = Invalid
client ID irep-polperfdc01.ca.com:dd2...426319724f88} |
org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ
BrokerService[dc_broker_dd2413a4-c930-4082-be05-426319724f88] Task-299

Broker-2

2015-07-27 00:35:25,203 | INFO  | Network connection between
vm://da_broker#12 and tcp:///138.42.106.19:37811@61616 shutdown due to a
local error: javax.jms.ResourceAllocationException: Temp Store is Full (6%
of 10737418240). Stopping producer
(dc_broker_dd2413a4-c930-4082-be05-426319724f88->da_broker-47643-1437954343579-4:2:1:1)
to prevent flooding
queue://DIP-poll.responses.irep-polperfdc01.ca.com:dd2413a4-c930-4082-be05-426319724f88.
See http://activemq.apache.org/producer-flow-control.html for more info |
org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ
Transport: tcp:///138.42.106.19:37811@61616


Any idea what's going on?

Thanks!
-Yang






--
View this message in context: http://activemq.2283324.n4.nabble.com/broker-network-disconnected-30mins-after-producer-s-tempUsage-full-tp4699899.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.