You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by terrytriple <te...@edinburgh.net.uk> on 2006/11/24 17:42:48 UTC
Multicast / Application Level Multicast / JXTA ?
Hi There,
Currently we are using JBossMQ to send data in a pub/sub model to a few
consumers,
over the internet - our testing has shown ActiveMQ to be significantly
faster than JBossMQ :
600-700 2k messages / minute with JBossMQ and 1,600 2k messages / minute
with ActiveMQ.
- 1 Publisher (pretty good / fast machine - 2 x 3GHz, 2GB Ram, RAID 10,
etc)
- Publisher in on a T1 Line
- Consumers are on DSL
- Sending messages over the internet
- Messages are Persistant, and Transactional
we need to sustain at least 200 messages / minute per client, the above
figures are an aggregate figure,
e.g. 600-700 m/m with JBossMQ equates to 200-240 m/m for 3 consumers,
However - we need this to upscale this solution from 4-5 consumers to 200
consumers,
very soon -- and based on the tests we have done - we would need to invest
in lots of
bandwith and several servers to get this to work.
e.g. 200 consumers x 200 m/m = 40,000 m/m total
so -- this being the case - we are looking for alternative solutions,
we have been looking at using multicast over the internet,
but information seems sketchy,
and although I have tried speaking to many ISPs about it
- It's difficult to get anyone who knows what I'm talking about,
and some simply say no.
besides the fact it seems AMQ 4 does not seem to support multicast delivery
? (only discovery).
would multicast via JGroups or JXTA work ? do these work ?
I believe JXTA would/could do some sort of Virtual / Application Level
Multicast and perhaps organise the nodes into a tree and delivery messages
that way ?
we like the idea of using a "Network of Brokers", or setting up some kind of
tree,
where our central server sends the messages to a few other brokers, and then
more brokers collect from those.
but we don't really want to have to go round each server and manually
configure the connections.
could the configuration be made more dynamic ? would it be possible to send
configuration changes via JMS ?
Anyone got any advice / thoughts ?
Many Thanks
--
Terry Paterson
Actual Systems UK Ltd
--
View this message in context: http://www.nabble.com/Multicast---Application-Level-Multicast---JXTA---tf2699960.html#a7528866
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Multicast / Application Level Multicast / JXTA ?
Posted by James Strachan <ja...@gmail.com>.
On 11/25/06, terrytriple <te...@edinburgh.net.uk> wrote:
> What we have been seeing with our testing is a linear decrease in delivery
> rate per consumer as we increase the number of consumers
As I said, for one producer, use async dispatch. The default of
ActiveMQ is for typical high performance scenario of multiple
producers and for maximum throughput we use sync dispatch out of the
box.
http://incubator.apache.org/activemq/consumer-dispatch-async.html
> e.g. 1,600 m/m with 1 consumer, 800 m/m with 2, 400 m/m with 4, 320 m/m with
> 5, etc..
>
> Note that these messages are being delivered over the internet,
> and we are using Pub/Sub with Topics,
> The producer machine is a pretty nice/quick machine - and we don't see any
> bottlenecks there,
> the bottleneck seems to be in our T1 connection.
>
> but to make this scale to 200-250 consumers, and maintain as least 200 m/m
> per consumer - it seems we would need A LOT of bandwidth !
>
> hence the reason for looking at multicast / network of brokers / JXTA /
> etc..
>
> I thought _maybe_ JXTA and its PIPEs could be configured in such a way as to
> give a psuedo/virtual multicast using a tree of nodes.
>
> or do the same thing with ActiveMQ brokers -- but again -- we don't want to
> have to setup every node by hand.
>
>
> any other ideas ?
If saving bandwidth is a concern you could look at some kind of tree
based system with store & forward across multiple brokers - assuming
that there is some way to collocate brokers nearer to consumers to
save bandwidth sending to one broker who fans out to multiple
consumers (otherwise there's no point). What is the network topology?
--
James
-------
http://radio.weblogs.com/0112098/
Re: Multicast / Application Level Multicast / JXTA ?
Posted by terrytriple <te...@edinburgh.net.uk>.
What we have been seeing with our testing is a linear decrease in delivery
rate per consumer as we increase the number of consumers
e.g. 1,600 m/m with 1 consumer, 800 m/m with 2, 400 m/m with 4, 320 m/m with
5, etc..
Note that these messages are being delivered over the internet,
and we are using Pub/Sub with Topics,
The producer machine is a pretty nice/quick machine - and we don't see any
bottlenecks there,
the bottleneck seems to be in our T1 connection.
but to make this scale to 200-250 consumers, and maintain as least 200 m/m
per consumer - it seems we would need A LOT of bandwidth !
hence the reason for looking at multicast / network of brokers / JXTA /
etc..
I thought _maybe_ JXTA and its PIPEs could be configured in such a way as to
give a psuedo/virtual multicast using a tree of nodes.
or do the same thing with ActiveMQ brokers -- but again -- we don't want to
have to setup every node by hand.
any other ideas ?
James.Strachan wrote:
>
> Am not aware of anyone who uses multicast over the internet - even
> JXTA is mostly socket based AFAIK, it just deals with NAT issues I
> thought but using regular sockets?
>
> So I'd just use a couple of brokers. I guess you're using topics
> right? A single broker should be able to handle 1000s of clients with
> hundreds of messages a second per consumer on a fast network & decent
> PC with a reasonable disk.
>
> If you've only 1 producer, you might wanna use async dispatch to get
> the best throughput across hundreds of clients for a single producer.
>
> What kind of throughput do you get in your environment when you try
> hundreds of clients?
>
>
> On 11/24/06, terrytriple <te...@edinburgh.net.uk> wrote:
>>
>> Hi There,
>>
>> Currently we are using JBossMQ to send data in a pub/sub model to a few
>> consumers,
>> over the internet - our testing has shown ActiveMQ to be significantly
>> faster than JBossMQ :
>>
>> 600-700 2k messages / minute with JBossMQ and 1,600 2k messages / minute
>> with ActiveMQ.
>> - 1 Publisher (pretty good / fast machine - 2 x 3GHz, 2GB Ram, RAID 10,
>> etc)
>> - Publisher in on a T1 Line
>> - Consumers are on DSL
>> - Sending messages over the internet
>> - Messages are Persistant, and Transactional
>>
>> we need to sustain at least 200 messages / minute per client, the above
>> figures are an aggregate figure,
>> e.g. 600-700 m/m with JBossMQ equates to 200-240 m/m for 3 consumers,
>>
>> However - we need this to upscale this solution from 4-5 consumers to 200
>> consumers,
>> very soon -- and based on the tests we have done - we would need to
>> invest
>> in lots of
>> bandwith and several servers to get this to work.
>>
>> e.g. 200 consumers x 200 m/m = 40,000 m/m total
>>
>> so -- this being the case - we are looking for alternative solutions,
>>
>> we have been looking at using multicast over the internet,
>> but information seems sketchy,
>> and although I have tried speaking to many ISPs about it
>> - It's difficult to get anyone who knows what I'm talking about,
>> and some simply say no.
>>
>> besides the fact it seems AMQ 4 does not seem to support multicast
>> delivery
>> ? (only discovery).
>>
>> would multicast via JGroups or JXTA work ? do these work ?
>>
>> I believe JXTA would/could do some sort of Virtual / Application Level
>> Multicast and perhaps organise the nodes into a tree and delivery
>> messages
>> that way ?
>>
>> we like the idea of using a "Network of Brokers", or setting up some kind
>> of
>> tree,
>> where our central server sends the messages to a few other brokers, and
>> then
>> more brokers collect from those.
>> but we don't really want to have to go round each server and manually
>> configure the connections.
>>
>> could the configuration be made more dynamic ? would it be possible to
>> send
>> configuration changes via JMS ?
>>
>> Anyone got any advice / thoughts ?
>>
>>
>> Many Thanks
>>
>> --
>> Terry Paterson
>> Actual Systems UK Ltd
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Multicast---Application-Level-Multicast---JXTA---tf2699960.html#a7528866
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>>
>
>
> --
>
> James
> -------
> http://radio.weblogs.com/0112098/
>
>
--
View this message in context: http://www.nabble.com/Multicast---Application-Level-Multicast---JXTA---tf2699960.html#a7540845
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Multicast / Application Level Multicast / JXTA ?
Posted by James Strachan <ja...@gmail.com>.
Am not aware of anyone who uses multicast over the internet - even
JXTA is mostly socket based AFAIK, it just deals with NAT issues I
thought but using regular sockets?
So I'd just use a couple of brokers. I guess you're using topics
right? A single broker should be able to handle 1000s of clients with
hundreds of messages a second per consumer on a fast network & decent
PC with a reasonable disk.
If you've only 1 producer, you might wanna use async dispatch to get
the best throughput across hundreds of clients for a single producer.
What kind of throughput do you get in your environment when you try
hundreds of clients?
On 11/24/06, terrytriple <te...@edinburgh.net.uk> wrote:
>
> Hi There,
>
> Currently we are using JBossMQ to send data in a pub/sub model to a few
> consumers,
> over the internet - our testing has shown ActiveMQ to be significantly
> faster than JBossMQ :
>
> 600-700 2k messages / minute with JBossMQ and 1,600 2k messages / minute
> with ActiveMQ.
> - 1 Publisher (pretty good / fast machine - 2 x 3GHz, 2GB Ram, RAID 10,
> etc)
> - Publisher in on a T1 Line
> - Consumers are on DSL
> - Sending messages over the internet
> - Messages are Persistant, and Transactional
>
> we need to sustain at least 200 messages / minute per client, the above
> figures are an aggregate figure,
> e.g. 600-700 m/m with JBossMQ equates to 200-240 m/m for 3 consumers,
>
> However - we need this to upscale this solution from 4-5 consumers to 200
> consumers,
> very soon -- and based on the tests we have done - we would need to invest
> in lots of
> bandwith and several servers to get this to work.
>
> e.g. 200 consumers x 200 m/m = 40,000 m/m total
>
> so -- this being the case - we are looking for alternative solutions,
>
> we have been looking at using multicast over the internet,
> but information seems sketchy,
> and although I have tried speaking to many ISPs about it
> - It's difficult to get anyone who knows what I'm talking about,
> and some simply say no.
>
> besides the fact it seems AMQ 4 does not seem to support multicast delivery
> ? (only discovery).
>
> would multicast via JGroups or JXTA work ? do these work ?
>
> I believe JXTA would/could do some sort of Virtual / Application Level
> Multicast and perhaps organise the nodes into a tree and delivery messages
> that way ?
>
> we like the idea of using a "Network of Brokers", or setting up some kind of
> tree,
> where our central server sends the messages to a few other brokers, and then
> more brokers collect from those.
> but we don't really want to have to go round each server and manually
> configure the connections.
>
> could the configuration be made more dynamic ? would it be possible to send
> configuration changes via JMS ?
>
> Anyone got any advice / thoughts ?
>
>
> Many Thanks
>
> --
> Terry Paterson
> Actual Systems UK Ltd
>
> --
> View this message in context: http://www.nabble.com/Multicast---Application-Level-Multicast---JXTA---tf2699960.html#a7528866
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
--
James
-------
http://radio.weblogs.com/0112098/