You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by anoppe <au...@noppe.nl> on 2017/04/24 08:04:46 UTC
Network of brokers and forwarding performance
Hi everybody,
I've setup 2 brokers with different responsibilities:
mq1 is an instance the webapplications consumes messages from and is
receiving messages also
mq2 is an instance that received messages only.
mq2 has some composite topics configured:
<destinationInterceptors>
<virtualDestinationInterceptor>
<virtualDestinations>
<compositeTopic name="company.*.topic" forwardOnly="false">
<forwardTo>
<topic physicalName="event.stream.topic" />
</forwardTo>
</compositeTopic>
<compositeTopic name="data.stream.topic" forwardOnly="false">
<forwardTo>
<topic physicalName="event.stream.topic" />
</forwardTo>
</compositeTopic>
<compositeTopic name="alert.triggers.forwarder.topic"
forwardOnly="true">
<forwardTo>
<topic physicalName="alert.triggers.topic" />
</forwardTo>
</compositeTopic>
</virtualDestinations>
</virtualDestinationInterceptor>
</destinationInterceptors>
- All topics that match the name 'company.*.topic' are duplicated to
'event.stream.topic'
- The topic 'data.stream.topic' is duplicated to 'event.stream.topic'
- The topic 'alert.triggers.forwarder.topic' is duplicated to
'alert.triggers.topic'.
mq1 and mq2 are in a network of brokers, where mq2 is forwarding the
messages in 'event.stream.topic' and 'alert.triggers.topic' to mq1, so the
consumers on that instance receive those messages also. To prevent
duplication, I've excluded 'data.stream.topic' form the network-of-brokers:
<networkConnectors>
<networkConnector name="eventStreamAndAlertTriggersForwarder"
uri="static:(tcp://ip-of-mq1:61616)" duplex="false">
<excludedDestinations>
<topic physicalName="data.stream.topic" />
</excludedDestinations>
<staticallyIncludedDestinations>
<topic physicalName="event.stream.topic"/>
<topic physicalName="alert.triggers.topic"/>
</staticallyIncludedDestinations>
</networkConnector>
</networkConnectors>
On a functional level this works just fine. But, we tried to loadtest our
infrastructure and ran into unexpected behaviour: When we send messages at a
rate of ~200m/s to 'company.*.topic'@mq2, consuming from mq2 becomes very
slow: When we stop sending new messages, we are able to consume at a rate of
~30m/s. If we start sending messages again, the rate of consuming drops to
~5m/s and a queue builds up.
The consumer we measured runs on the same machine.
When we send and consume messages on mq1, we get the expected rate of about
30m/s.
Does anyone has an idea how we can improve performance? Did I misconfigure
something?
Regards,
Auke Noppe
--
View this message in context: http://activemq.2283324.n4.nabble.com/Network-of-brokers-and-forwarding-performance-tp4725187.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Network of brokers and forwarding performance
Posted by Tim Bain <tb...@alumni.duke.edu>.
Auke,
Pooling your connections is definitely a good thing, but since you say that
the system still exhibits the performance slowdown, your next step is what
I said in my first response: to characterize which OS resource (CPU,
network, disk, memory) is the limiting one, and on which broker or client
process.
Thanks for the detailed description of how you're using the network of
brokers; that clears up my confusion.
Tim
On Apr 25, 2017 8:10 AM, "anoppe" <au...@noppe.nl> wrote:
> Hi Tim,Thank you for your replies.I've found one bug in our software: We
> didn't used a PooledConnectionFactory, which caused every message to be
> produced to mq2 setup a new connection to mq2. I've fixed this in the
> meanwhile and repeated my tests; the results were somewhat better (+1m/s),
> but not satisfying enough.To answer your 'BTW' reply: We have split the
> message-flow into two separate parts: mq1 and mq2. We have an API where
> clients sent messages to, which gets published to either mq1 or mq2. The
> responsibility of mq1 is to process 'legacy' messages from the API, and
> mq2's responsibility is to process 'new' messages from the API. The
> datamodel of those messages are exactly the same, but we choose to split
> into separate flows to minimise interference for existing customers.The
> reason we introduced the 'network of brokers'-pattern:We have a number of
> other web-applications/API's that consume messages from 'even.stream.topic'
> (is an aggregate of all topics om mq2 and mq1) from mq1. We didn't want to
> change all these applications to consume from both brokers, therefore we
> decided to let mq2 forward 'company.*.topic'@mq2 contents to mq1. (And
> some
> other topics, but are not relevant for the concept, in my opinion)Recap:-
> All topics that match the name 'company.*.topic'@mq2 are duplicated to
> 'event.stream.topic'@mq1 - The topic 'data.stream.topic'@mq2 is duplicated
> to 'event.stream.topic'@mq1 - The topic 'alert.triggers.forwarder.
> topic'@mq2
> is duplicated to 'alert.triggers.topic'@mq1. - When testing, I publish
> messages to mq2 that will end up in only 1 specific 'company.*.topic'.-
> That
> specific 'company.*.topic'@mq2 gets consumed by an application that
> inserts
> that message into the database, which I measure the consuming-rate on.
> (5m/s)- Another instance of that application is consuming from mq1. (which
> consumes ~30m/s)- If I stop producing measurements on mq2, the rate of
> consuming increases to ~20-25m/s. Which is acceptable.Does this help in
> understanding our situation and the problem we're facing?Regards,Auke Noppe
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/Network-of-brokers-and-forwarding-performance-
> tp4725187p4725223.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Network of brokers and forwarding performance
Posted by anoppe <au...@noppe.nl>.
Hi Tim,Thank you for your replies.I've found one bug in our software: We
didn't used a PooledConnectionFactory, which caused every message to be
produced to mq2 setup a new connection to mq2. I've fixed this in the
meanwhile and repeated my tests; the results were somewhat better (+1m/s),
but not satisfying enough.To answer your 'BTW' reply: We have split the
message-flow into two separate parts: mq1 and mq2. We have an API where
clients sent messages to, which gets published to either mq1 or mq2. The
responsibility of mq1 is to process 'legacy' messages from the API, and
mq2's responsibility is to process 'new' messages from the API. The
datamodel of those messages are exactly the same, but we choose to split
into separate flows to minimise interference for existing customers.The
reason we introduced the 'network of brokers'-pattern:We have a number of
other web-applications/API's that consume messages from 'even.stream.topic'
(is an aggregate of all topics om mq2 and mq1) from mq1. We didn't want to
change all these applications to consume from both brokers, therefore we
decided to let mq2 forward 'company.*.topic'@mq2 contents to mq1. (And some
other topics, but are not relevant for the concept, in my opinion)Recap:-
All topics that match the name 'company.*.topic'@mq2 are duplicated to
'event.stream.topic'@mq1 - The topic 'data.stream.topic'@mq2 is duplicated
to 'event.stream.topic'@mq1 - The topic 'alert.triggers.forwarder.topic'@mq2
is duplicated to 'alert.triggers.topic'@mq1. - When testing, I publish
messages to mq2 that will end up in only 1 specific 'company.*.topic'.- That
specific 'company.*.topic'@mq2 gets consumed by an application that inserts
that message into the database, which I measure the consuming-rate on.
(5m/s)- Another instance of that application is consuming from mq1. (which
consumes ~30m/s)- If I stop producing measurements on mq2, the rate of
consuming increases to ~20-25m/s. Which is acceptable.Does this help in
understanding our situation and the problem we're facing?Regards,Auke Noppe
--
View this message in context: http://activemq.2283324.n4.nabble.com/Network-of-brokers-and-forwarding-performance-tp4725187p4725223.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Network of brokers and forwarding performance
Posted by Tim Bain <tb...@alumni.duke.edu>.
BTW, I thought from your description that mq2 was a publish-only node, but
you said you're testing performance by consuming from it. Can you please
describe again how you're using the nodes, using the words "publish" and
"consume" instead of the ambiguous word "receive"?
Tim
On Apr 25, 2017 7:14 AM, "Tim Bain" <tb...@alumni.duke.edu> wrote:
> Quoting from an email I posted in another thread a few days ago:
>
> When a system is slow, there's always something (or several somethings,
> rotating between them) that is the limiting resource. I'd start by trying
> to determine what that resource is, using standard OS monitoring tools (top
> and the like). If you can tell that CPU is a primary bottleneck, use a
> profiler or sampler like JVisualVM to determine where the time is being
> spent, and share what you find here (maybe there's a bug we need to fix).
> If it's disk IO, make sure your Log4J logging is at the right level (i.e.
> not DEBUG or TRACE), and if it's still the bottleneck, consider options to
> upgrade your storage hardware to something more performant. If the JVM is
> spending a lot of time garbage collecting (JVisualVM and JConsole will tell
> you that), increase the broker's heap size. If your OS is heavily into
> swap, buy more RAM. And the list goes on, but the starting point is you
> characterizing what's actually slow, and we can help you figure out why
> it's slow.
>
> Tim
>
> On Apr 24, 2017 2:15 AM, "anoppe" <au...@noppe.nl> wrote:
>
>> Hi everybody,
>>
>> I've setup 2 brokers with different responsibilities:
>> mq1 is an instance the webapplications consumes messages from and is
>> receiving messages also
>> mq2 is an instance that received messages only.
>>
>> mq2 has some composite topics configured:
>> <destinationInterceptors>
>> <virtualDestinationInterceptor>
>> <virtualDestinations>
>> <compositeTopic name="company.*.topic" forwardOnly="false">
>> <forwardTo>
>> <topic physicalName="event.stream.topic" />
>> </forwardTo>
>> </compositeTopic>
>> <compositeTopic name="data.stream.topic"
>> forwardOnly="false">
>> <forwardTo>
>> <topic physicalName="event.stream.topic" />
>> </forwardTo>
>> </compositeTopic>
>> <compositeTopic name="alert.triggers.forwarder.topic"
>> forwardOnly="true">
>> <forwardTo>
>> <topic physicalName="alert.triggers.topic" />
>> </forwardTo>
>> </compositeTopic>
>> </virtualDestinations>
>> </virtualDestinationInterceptor>
>> </destinationInterceptors>
>>
>> - All topics that match the name 'company.*.topic' are duplicated to
>> 'event.stream.topic'
>> - The topic 'data.stream.topic' is duplicated to 'event.stream.topic'
>> - The topic 'alert.triggers.forwarder.topic' is duplicated to
>> 'alert.triggers.topic'.
>>
>> mq1 and mq2 are in a network of brokers, where mq2 is forwarding the
>> messages in 'event.stream.topic' and 'alert.triggers.topic' to mq1, so the
>> consumers on that instance receive those messages also. To prevent
>> duplication, I've excluded 'data.stream.topic' form the
>> network-of-brokers:
>> <networkConnectors>
>> <networkConnector name="eventStreamAndAlertTriggersForwarder"
>> uri="static:(tcp://ip-of-mq1:61616)" duplex="false">
>> <excludedDestinations>
>> <topic physicalName="data.stream.topic" />
>> </excludedDestinations>
>> <staticallyIncludedDestinations>
>> <topic physicalName="event.stream.topic"/>
>> <topic physicalName="alert.triggers.topic"/>
>> </staticallyIncludedDestinations>
>> </networkConnector>
>> </networkConnectors>
>>
>> On a functional level this works just fine. But, we tried to loadtest our
>> infrastructure and ran into unexpected behaviour: When we send messages
>> at a
>> rate of ~200m/s to 'company.*.topic'@mq2, consuming from mq2 becomes very
>> slow: When we stop sending new messages, we are able to consume at a rate
>> of
>> ~30m/s. If we start sending messages again, the rate of consuming drops to
>> ~5m/s and a queue builds up.
>> The consumer we measured runs on the same machine.
>> When we send and consume messages on mq1, we get the expected rate of
>> about
>> 30m/s.
>>
>> Does anyone has an idea how we can improve performance? Did I misconfigure
>> something?
>>
>> Regards,
>> Auke Noppe
>>
>>
>>
>>
>>
>> --
>> View this message in context: http://activemq.2283324.n4.nab
>> ble.com/Network-of-brokers-and-forwarding-performance-tp4725187.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>
Re: Network of brokers and forwarding performance
Posted by Tim Bain <tb...@alumni.duke.edu>.
Quoting from an email I posted in another thread a few days ago:
When a system is slow, there's always something (or several somethings,
rotating between them) that is the limiting resource. I'd start by trying
to determine what that resource is, using standard OS monitoring tools (top
and the like). If you can tell that CPU is a primary bottleneck, use a
profiler or sampler like JVisualVM to determine where the time is being
spent, and share what you find here (maybe there's a bug we need to fix).
If it's disk IO, make sure your Log4J logging is at the right level (i.e.
not DEBUG or TRACE), and if it's still the bottleneck, consider options to
upgrade your storage hardware to something more performant. If the JVM is
spending a lot of time garbage collecting (JVisualVM and JConsole will tell
you that), increase the broker's heap size. If your OS is heavily into
swap, buy more RAM. And the list goes on, but the starting point is you
characterizing what's actually slow, and we can help you figure out why
it's slow.
Tim
On Apr 24, 2017 2:15 AM, "anoppe" <au...@noppe.nl> wrote:
> Hi everybody,
>
> I've setup 2 brokers with different responsibilities:
> mq1 is an instance the webapplications consumes messages from and is
> receiving messages also
> mq2 is an instance that received messages only.
>
> mq2 has some composite topics configured:
> <destinationInterceptors>
> <virtualDestinationInterceptor>
> <virtualDestinations>
> <compositeTopic name="company.*.topic" forwardOnly="false">
> <forwardTo>
> <topic physicalName="event.stream.topic" />
> </forwardTo>
> </compositeTopic>
> <compositeTopic name="data.stream.topic" forwardOnly="false">
> <forwardTo>
> <topic physicalName="event.stream.topic" />
> </forwardTo>
> </compositeTopic>
> <compositeTopic name="alert.triggers.forwarder.topic"
> forwardOnly="true">
> <forwardTo>
> <topic physicalName="alert.triggers.topic" />
> </forwardTo>
> </compositeTopic>
> </virtualDestinations>
> </virtualDestinationInterceptor>
> </destinationInterceptors>
>
> - All topics that match the name 'company.*.topic' are duplicated to
> 'event.stream.topic'
> - The topic 'data.stream.topic' is duplicated to 'event.stream.topic'
> - The topic 'alert.triggers.forwarder.topic' is duplicated to
> 'alert.triggers.topic'.
>
> mq1 and mq2 are in a network of brokers, where mq2 is forwarding the
> messages in 'event.stream.topic' and 'alert.triggers.topic' to mq1, so the
> consumers on that instance receive those messages also. To prevent
> duplication, I've excluded 'data.stream.topic' form the network-of-brokers:
> <networkConnectors>
> <networkConnector name="eventStreamAndAlertTriggersForwarder"
> uri="static:(tcp://ip-of-mq1:61616)" duplex="false">
> <excludedDestinations>
> <topic physicalName="data.stream.topic" />
> </excludedDestinations>
> <staticallyIncludedDestinations>
> <topic physicalName="event.stream.topic"/>
> <topic physicalName="alert.triggers.topic"/>
> </staticallyIncludedDestinations>
> </networkConnector>
> </networkConnectors>
>
> On a functional level this works just fine. But, we tried to loadtest our
> infrastructure and ran into unexpected behaviour: When we send messages at
> a
> rate of ~200m/s to 'company.*.topic'@mq2, consuming from mq2 becomes very
> slow: When we stop sending new messages, we are able to consume at a rate
> of
> ~30m/s. If we start sending messages again, the rate of consuming drops to
> ~5m/s and a queue builds up.
> The consumer we measured runs on the same machine.
> When we send and consume messages on mq1, we get the expected rate of about
> 30m/s.
>
> Does anyone has an idea how we can improve performance? Did I misconfigure
> something?
>
> Regards,
> Auke Noppe
>
>
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/Network-of-brokers-and-forwarding-performance-tp4725187.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>