Network Connector Performance

[Ehud Eshet <eh...@imperva.com>]

Hi,

I have a producer that must be able to keep producing even when the remote
broker is unavailable.
failover is not enough since the problem might be in the network or even the
local NIC.

Thus, I planed to have a local broker that will use a network connector to
route all messages to the remote broker.

I use ActiveMQ 5.10 with levelDB persistence on Java SE 1.7 (both local and
remote servers).
Average message length is 1KB.
I need to be able to send at least 10,000 message per second.

Send throughput to the local broker is about 100 messages per second.
When using transactions and committing every 1000 messages, throughput jump
to 12,000 messages per second.

The network connector doesn't use transactions.
Thus, its throughput is only 100 messages per second.

I tried to use proxy connector in order to force useAsyncSend=true on the
network connector with no success.

Finally, I removed the network connector all together and implemented its
functionality myself.
In another thread on the local server I used a transacted session to local
broker plus a transacted session to the remote broker. Each message received
from the local broker was sent into the remote broker. Every 1000 messages
(or when more than a second passed since last commit), I committed both
sessions.

The throughput of the new forwarder thread is 11,000 message per second.
When running in parallel to the local producer the throughput is still
11,000 messages per second.

I would like to suggest two additional properties to networkConnector:
1. maxBatchSize - if specified and higher than 1, use transactions to
forward messages to remote broker.
2. maxLatency.

The network connector will send (commit)  up to <maxBatchSize> messages in a
transaction.
It will commit immediately, if more than <maxLatency> milliseconds passed
since last commit. 



--
View this message in context: http://activemq.2283324.n4.nabble.com/Network-Connector-Performance-tp4690186.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Network Connector Performance

[Tim Bain <tb...@alumni.duke.edu>]

As long as no one objects to the idea in the next day or two, you should
submit this enhancement request via JIRA (
https://issues.apache.org/jira/browse/AMQ).
On Jan 21, 2015 1:07 AM, "Ehud Eshet" <eh...@imperva.com> wrote:

> Hi,
>
> I have a producer that must be able to keep producing even when the remote
> broker is unavailable.
> failover is not enough since the problem might be in the network or even
> the
> local NIC.
>
> Thus, I planed to have a local broker that will use a network connector to
> route all messages to the remote broker.
>
> I use ActiveMQ 5.10 with levelDB persistence on Java SE 1.7 (both local and
> remote servers).
> Average message length is 1KB.
> I need to be able to send at least 10,000 message per second.
>
> Send throughput to the local broker is about 100 messages per second.
> When using transactions and committing every 1000 messages, throughput jump
> to 12,000 messages per second.
>
> The network connector doesn't use transactions.
> Thus, its throughput is only 100 messages per second.
>
> I tried to use proxy connector in order to force useAsyncSend=true on the
> network connector with no success.
>
> Finally, I removed the network connector all together and implemented its
> functionality myself.
> In another thread on the local server I used a transacted session to local
> broker plus a transacted session to the remote broker. Each message
> received
> from the local broker was sent into the remote broker. Every 1000 messages
> (or when more than a second passed since last commit), I committed both
> sessions.
>
> The throughput of the new forwarder thread is 11,000 message per second.
> When running in parallel to the local producer the throughput is still
> 11,000 messages per second.
>
> I would like to suggest two additional properties to networkConnector:
> 1. maxBatchSize - if specified and higher than 1, use transactions to
> forward messages to remote broker.
> 2. maxLatency.
>
> The network connector will send (commit)  up to <maxBatchSize> messages in
> a
> transaction.
> It will commit immediately, if more than <maxLatency> milliseconds passed
> since last commit.
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Network-Connector-Performance-tp4690186.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>