You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by shital <sh...@optonline.net> on 2007/01/08 21:24:10 UTC
High Volume of messages
Hi !!!
I have an extremely high volume of messages. Several thousands of messages
per second. It does not care for message drops, and we use async dispatch to
get maximum performance. Now my confusion is that if I use 3 brokers to make
a network, and use failover protocol, will the producer have to worry about
making sure that messages are transferred to all 3 brokers in the cluster?
Also since such a high volume of messages is transferred across 3 brokers,
would it slow down the whole consumer/producer chain ??
--
View this message in context: http://www.nabble.com/High-Volume-of-messages-tf2941814.html#a8225745
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: High Volume of messages
Posted by Rob Davies <ra...@gmail.com>.
In that case - if you've a good test case you could submit, I'm sure
you'll get a lot of folks (me included) willing to tune ActiveMq for
this scenario
cheers,
Rob
On 12 Jan 2007, at 13:17, Abdul Alhazred wrote:
>
> Yeah, I have exactly that scenario ;)
>
>
> James.Strachan wrote:
>>
>> On 1/11/07, Abdul Alhazred <th...@tradedesksoftware.com> wrote:
>>> Interesting, so what about using a multicast transport with a
>>> network of
>>> brokers? If all the brokers can see the multicast then they
>>> shouldn't
>>> need
>>> store and forward at all.
>>
>> I'd recommend TCP if you want reliability. Currently we tend to use
>> multicast purely for discovery as if a node misses a message due to
>> packet loss (or the input buffer being full so the network card
>> discards it) then a point-to-point redelivery mechanism is required
>> which can cause network storms if you're not careful. We've got the
>> start of a basic implentation of reliable UDP/multicast but its not
>> completely flushed out and working just yet.
>>
>> If the messages are small (so they fit inside a single datagram) and
>> you don't care about loosing them (such as for market data prices) by
>> all means give multicast a try if you want.
>>
>> --
>>
>> James
>> -------
>> http://radio.weblogs.com/0112098/
>>
>>
>
> --
> View this message in context: http://www.nabble.com/High-Volume-of-
> messages-tf2941814.html#a8298155
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: High Volume of messages
Posted by Abdul Alhazred <th...@tradedesksoftware.com>.
Yeah, I have exactly that scenario ;)
James.Strachan wrote:
>
> On 1/11/07, Abdul Alhazred <th...@tradedesksoftware.com> wrote:
>> Interesting, so what about using a multicast transport with a network of
>> brokers? If all the brokers can see the multicast then they shouldn't
>> need
>> store and forward at all.
>
> I'd recommend TCP if you want reliability. Currently we tend to use
> multicast purely for discovery as if a node misses a message due to
> packet loss (or the input buffer being full so the network card
> discards it) then a point-to-point redelivery mechanism is required
> which can cause network storms if you're not careful. We've got the
> start of a basic implentation of reliable UDP/multicast but its not
> completely flushed out and working just yet.
>
> If the messages are small (so they fit inside a single datagram) and
> you don't care about loosing them (such as for market data prices) by
> all means give multicast a try if you want.
>
> --
>
> James
> -------
> http://radio.weblogs.com/0112098/
>
>
--
View this message in context: http://www.nabble.com/High-Volume-of-messages-tf2941814.html#a8298155
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: High Volume of messages
Posted by James Strachan <ja...@gmail.com>.
On 1/11/07, Abdul Alhazred <th...@tradedesksoftware.com> wrote:
> Interesting, so what about using a multicast transport with a network of
> brokers? If all the brokers can see the multicast then they shouldn't need
> store and forward at all.
I'd recommend TCP if you want reliability. Currently we tend to use
multicast purely for discovery as if a node misses a message due to
packet loss (or the input buffer being full so the network card
discards it) then a point-to-point redelivery mechanism is required
which can cause network storms if you're not careful. We've got the
start of a basic implentation of reliable UDP/multicast but its not
completely flushed out and working just yet.
If the messages are small (so they fit inside a single datagram) and
you don't care about loosing them (such as for market data prices) by
all means give multicast a try if you want.
--
James
-------
http://radio.weblogs.com/0112098/
Re: High Volume of messages
Posted by Abdul Alhazred <th...@tradedesksoftware.com>.
Interesting, so what about using a multicast transport with a network of
brokers? If all the brokers can see the multicast then they shouldn't need
store and forward at all.
James.Strachan wrote:
>
> Agreed. We've also got the fanout transport (hopefully being souped up
> into the Jedi transport some day - John? :), which will allow a JMS
> client to in parallel send a message to multiple brokers which can
> help reduce latency and provides an alternative with topics to using
> broker store & forward networks.
>
>
> On 1/9/07, Abdul Alhazred <th...@tradedesksoftware.com> wrote:
>>
>> No, with clustering you don't have to do anything special at the client.
>> Basically the brokers will forward messages as necessary to reach
>> whichever
>> consumers need to get them. So if Producer A is on broker 1 and sends a
>> message destined for Consumer B connected on broker 2, the message will
>> go
>> from A to 1 to 2 to B.
>>
>> That being said the topology of your network is going to determine the
>> amount of work going on. If every message has to go to consumers at all 3
>> brokers, then your brokers will pretty much be doing equal amounts of
>> work,
>> all of them will have to receive each message once and deliver it again
>> at
>> least one time. So it is possible such a network of brokers might not
>> perform much differently than 1 broker alone. OTOH some actions are
>> expensive, like persisting messages or using synchronous delivery.
>> Broker-broker communications can generally use the most efficient
>> mechanisms, so again it depends on exactly who uses what how.
>>
>> Benchmark. Nothing but a real test of a system is a solid answer to
>> performance questions. Run tests with one and with 3 and see what
>> happens.
>>
>>
>> shital wrote:
>> >
>> > Hi !!!
>> >
>> > I have an extremely high volume of messages. Several thousands of
>> messages
>> > per second. It does not care for message drops, and we use async
>> dispatch
>> > to get maximum performance. Now my confusion is that if I use 3 brokers
>> to
>> > make a network, and use failover protocol, will the producer have to
>> worry
>> > about making sure that messages are transferred to all 3 brokers in the
>> > cluster? Also since such a high volume of messages is transferred
>> across 3
>> > brokers, would it slow down the whole consumer/producer chain ??
>> >
>>
>> --
>> View this message in context:
>> http://www.nabble.com/High-Volume-of-messages-tf2941814.html#a8237114
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>>
>
>
> --
>
> James
> -------
> http://radio.weblogs.com/0112098/
>
>
--
View this message in context: http://www.nabble.com/High-Volume-of-messages-tf2941814.html#a8285282
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: High Volume of messages
Posted by James Strachan <ja...@gmail.com>.
Agreed. We've also got the fanout transport (hopefully being souped up
into the Jedi transport some day - John? :), which will allow a JMS
client to in parallel send a message to multiple brokers which can
help reduce latency and provides an alternative with topics to using
broker store & forward networks.
On 1/9/07, Abdul Alhazred <th...@tradedesksoftware.com> wrote:
>
> No, with clustering you don't have to do anything special at the client.
> Basically the brokers will forward messages as necessary to reach whichever
> consumers need to get them. So if Producer A is on broker 1 and sends a
> message destined for Consumer B connected on broker 2, the message will go
> from A to 1 to 2 to B.
>
> That being said the topology of your network is going to determine the
> amount of work going on. If every message has to go to consumers at all 3
> brokers, then your brokers will pretty much be doing equal amounts of work,
> all of them will have to receive each message once and deliver it again at
> least one time. So it is possible such a network of brokers might not
> perform much differently than 1 broker alone. OTOH some actions are
> expensive, like persisting messages or using synchronous delivery.
> Broker-broker communications can generally use the most efficient
> mechanisms, so again it depends on exactly who uses what how.
>
> Benchmark. Nothing but a real test of a system is a solid answer to
> performance questions. Run tests with one and with 3 and see what happens.
>
>
> shital wrote:
> >
> > Hi !!!
> >
> > I have an extremely high volume of messages. Several thousands of messages
> > per second. It does not care for message drops, and we use async dispatch
> > to get maximum performance. Now my confusion is that if I use 3 brokers to
> > make a network, and use failover protocol, will the producer have to worry
> > about making sure that messages are transferred to all 3 brokers in the
> > cluster? Also since such a high volume of messages is transferred across 3
> > brokers, would it slow down the whole consumer/producer chain ??
> >
>
> --
> View this message in context: http://www.nabble.com/High-Volume-of-messages-tf2941814.html#a8237114
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
--
James
-------
http://radio.weblogs.com/0112098/
Re: High Volume of messages
Posted by Abdul Alhazred <th...@tradedesksoftware.com>.
No, with clustering you don't have to do anything special at the client.
Basically the brokers will forward messages as necessary to reach whichever
consumers need to get them. So if Producer A is on broker 1 and sends a
message destined for Consumer B connected on broker 2, the message will go
from A to 1 to 2 to B.
That being said the topology of your network is going to determine the
amount of work going on. If every message has to go to consumers at all 3
brokers, then your brokers will pretty much be doing equal amounts of work,
all of them will have to receive each message once and deliver it again at
least one time. So it is possible such a network of brokers might not
perform much differently than 1 broker alone. OTOH some actions are
expensive, like persisting messages or using synchronous delivery.
Broker-broker communications can generally use the most efficient
mechanisms, so again it depends on exactly who uses what how.
Benchmark. Nothing but a real test of a system is a solid answer to
performance questions. Run tests with one and with 3 and see what happens.
shital wrote:
>
> Hi !!!
>
> I have an extremely high volume of messages. Several thousands of messages
> per second. It does not care for message drops, and we use async dispatch
> to get maximum performance. Now my confusion is that if I use 3 brokers to
> make a network, and use failover protocol, will the producer have to worry
> about making sure that messages are transferred to all 3 brokers in the
> cluster? Also since such a high volume of messages is transferred across 3
> brokers, would it slow down the whole consumer/producer chain ??
>
--
View this message in context: http://www.nabble.com/High-Volume-of-messages-tf2941814.html#a8237114
Sent from the ActiveMQ - User mailing list archive at Nabble.com.