[jira] [Commented] (AMQ-4971) OOM in DemandForwardingBridge

["Nikolay Martynov (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AMQ-4971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13873247#comment-13873247 ] 

Nikolay Martynov commented on AMQ-4971:
---------------------------------------

The issue can be easily reproduced in the following scenario:
1. Start 3 brokers. Let it be 0, 1, 2
2. In brokers 1 and 2 enable conduit duplex network connector to 0
3. Connect tcp:// producer for topic A to 0 (jmeter)
4. Connect vm:// consumer for topic A to 1 (simple jms message forwarder in camel: from jms, to jms)
5. Connect vm:// producer for topic B to 1 (simple jms message forwarder in camel: from jms, to jms)
6. Connect vm:// consumer for topic B to 2 (simple jms message forwarder in camel: from jms, to jms)
7. Connect vm:// producer for topic C to 2 (simple jms message forwarder in camel: from jms, to jms)
8. Connect tcp:// consumer for topic C to 0 (jmeter)
In summary, we just put bi-directional load to each of network bridges where each bridge has to both send and receive messages and acks.
Since vm:// and camel embedded in the same JVM are faster than tcp:// and peer broker, bridge very quickly fills heap with responseCallback's used to handle ack and reference message body. This isnt controlled by broker memory limits or producer flow control and bridges die on OOM.

Additionally, VMTransport will also keep references to asynchronously handles messages. Since vm:// is faster than tcp://, this is another source of quick OOM where producer flow control and memory limits have no any effect. While queue is bounded, default size is 2000 and with 1MB messages you quickly run out of heap (doc doesnt seem to explain if and how it could be adjusted). Wouldnt be a problem if it could be controlled by broker memory limits and producer flow control.

> OOM in DemandForwardingBridge
> -----------------------------
>
>                 Key: AMQ-4971
>                 URL: https://issues.apache.org/jira/browse/AMQ-4971
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Broker
>    Affects Versions: 5.9.0
>            Reporter: Yuriy Sidelnikov
>              Labels: features
>
> DemandForwardingBridge sends messages to the other broker and asynchronously waits for ACKs keeping message bodies in heap. Amount of un-ACK-ed messages kept in heap is not limited. If local producer is fast then whole heap will be consumed by messages waiting to be ACK-ed by other broker.
> Possible option to fix the issue:
> Don't wait for ACK from other broker when forwarding the message if some threshold of un-ACK-ed messages is reached.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)