You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Martin_Cornelius <Ma...@smiths-heimann.com> on 2008/09/05 19:10:31 UTC
Scalability of 'Networks of Brokers'
Hi all,
i am currently evaluating the scalability of Networks of ActiveMQ Brokers
with a large number of brokers. The actual drive behind this evaluation: I
have to design a distributed system that is completely serverless. It shall
be possible to switch of any of the machines in the system without alloying
the distributed functionality. The number of active machines in the system
might vary from only 2 to about 100.
My current idea: Have an (presumably embedded) ActiveMQ Broker running on
any of the machines, and let all this Brokers form a 'Network of Brokers'
(abbreviated NOB in the follwing)
So far, i found that in an NOB a message is forwarded only to those brokers
where at least one client has subscribed to the topic resp. queue the
message is sent to. Really fine. If a client subscribes to a queue, the
broker where the subscription is made sends a single short control-message
to all other brokers of the NOB. That would still allow for thousands of
brokers, fine again.
However, i just realized one fact that may compromise scalabilty: If a
client subscribes to a topic, the broker where the subscription is done
sends a not-so-short control-message to all other brokers. So far not too
bad, but now comes the problem: Every broker seems to react on this
control-message by sending another not-so-short control-message to all other
brokers. This means, if N brokers are present, every subscription to a topic
results in about (N-1)*N*2 control-messages (The additional factor two comes
from the fact, that the final recipient-broker seems to send another
response control-message). BTW, The length of the control-messages being
sent seems to be closely related to the string length of the topic name.
I have verified this behaviour with only up to 6 brokers, so i'm not totally
shure if it will extrapolate to a larger number, but i suspect it does. This
would mean, that with e.g. 100 Brokers every subscription to a topic would
result in 100*99*2 control messages being sent. If a single control message
has about 400 bytes (what i observed), this means a single subscription to a
topic would result in about 8 Megabyte network traffic. This might even be
sustainable, but the quadratic order kills scalabilty -- with 1000 Brokers
we would have to transfer about 1 Gigabyte, what is no longer feasible.
Finally, my questions: Is there any configuration option by which this
quadratic order can be avoided, is my idea simply foolish, or are my
conclusions wrong ? Note that i gained my conclusions more or less
'empirically' using wireshark.
thanks for any advice, Martin
--
View this message in context: http://www.nabble.com/Scalability-of-%27Networks-of-Brokers%27-tp19335699p19335699.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Scalability of 'Networks of Brokers'
Posted by "Thomas R. Corbin" <th...@gmail.com>.
On Friday 05 September 2008, Bryan Murphy said:
> Sounds like a nightmare to me. Maybe I don't understand what you're trying
> to accomplish, but I think something like Hadoop would be a better fit for
> this kind of problem.
sounds like a job that JINI was designed for.
>
> Bryan
>
> On Fri, Sep 5, 2008 at 12:10 PM, Martin_Cornelius <
>
> Martin.Cornelius@smiths-heimann.com> wrote:
> > Hi all,
> >
> > i am currently evaluating the scalability of Networks of ActiveMQ Brokers
> > with a large number of brokers. The actual drive behind this evaluation:
> > I have to design a distributed system that is completely serverless. It
> > shall be possible to switch of any of the machines in the system without
> > alloying the distributed functionality. The number of active machines in
> > the system might vary from only 2 to about 100.
> >
> > My current idea: Have an (presumably embedded) ActiveMQ Broker running on
> > any of the machines, and let all this Brokers form a 'Network of Brokers'
> > (abbreviated NOB in the follwing)
> >
> > So far, i found that in an NOB a message is forwarded only to those
> > brokers where at least one client has subscribed to the topic resp. queue
> > the message is sent to. Really fine. If a client subscribes to a queue,
> > the broker where the subscription is made sends a single short
> > control-message to all other brokers of the NOB. That would still allow
> > for thousands of brokers, fine again.
> >
> > However, i just realized one fact that may compromise scalabilty: If a
> > client subscribes to a topic, the broker where the subscription is done
> > sends a not-so-short control-message to all other brokers. So far not too
> > bad, but now comes the problem: Every broker seems to react on this
> > control-message by sending another not-so-short control-message to all
> > other
> > brokers. This means, if N brokers are present, every subscription to a
> > topic
> > results in about (N-1)*N*2 control-messages (The additional factor two
> > comes
> > from the fact, that the final recipient-broker seems to send another
> > response control-message). BTW, The length of the control-messages being
> > sent seems to be closely related to the string length of the topic name.
> >
> > I have verified this behaviour with only up to 6 brokers, so i'm not
> > totally
> > shure if it will extrapolate to a larger number, but i suspect it does.
> > This
> > would mean, that with e.g. 100 Brokers every subscription to a topic
> > would result in 100*99*2 control messages being sent. If a single control
> > message has about 400 bytes (what i observed), this means a single
> > subscription to a
> > topic would result in about 8 Megabyte network traffic. This might even
> > be sustainable, but the quadratic order kills scalabilty -- with 1000
> > Brokers we would have to transfer about 1 Gigabyte, what is no longer
> > feasible.
> >
> > Finally, my questions: Is there any configuration option by which this
> > quadratic order can be avoided, is my idea simply foolish, or are my
> > conclusions wrong ? Note that i gained my conclusions more or less
> > 'empirically' using wireshark.
> >
> > thanks for any advice, Martin
> >
> > --
> > View this message in context:
> > http://www.nabble.com/Scalability-of-%27Networks-of-Brokers%27-tp19335699
> >p19335699.html Sent from the ActiveMQ - User mailing list archive at
> > Nabble.com.
--
Card #1. Uncle Bob’s Three Laws
• Write no production code except to pass a failing test.
• Write only enough of a test to demonstrate a failure.
• Write only enough production code to pass the test.
Re: Scalability of 'Networks of Brokers'
Posted by Bryan Murphy <bm...@gmail.com>.
Sounds like a nightmare to me. Maybe I don't understand what you're trying
to accomplish, but I think something like Hadoop would be a better fit for
this kind of problem.
Bryan
On Fri, Sep 5, 2008 at 12:10 PM, Martin_Cornelius <
Martin.Cornelius@smiths-heimann.com> wrote:
>
> Hi all,
>
> i am currently evaluating the scalability of Networks of ActiveMQ Brokers
> with a large number of brokers. The actual drive behind this evaluation: I
> have to design a distributed system that is completely serverless. It shall
> be possible to switch of any of the machines in the system without alloying
> the distributed functionality. The number of active machines in the system
> might vary from only 2 to about 100.
>
> My current idea: Have an (presumably embedded) ActiveMQ Broker running on
> any of the machines, and let all this Brokers form a 'Network of Brokers'
> (abbreviated NOB in the follwing)
>
> So far, i found that in an NOB a message is forwarded only to those brokers
> where at least one client has subscribed to the topic resp. queue the
> message is sent to. Really fine. If a client subscribes to a queue, the
> broker where the subscription is made sends a single short control-message
> to all other brokers of the NOB. That would still allow for thousands of
> brokers, fine again.
>
> However, i just realized one fact that may compromise scalabilty: If a
> client subscribes to a topic, the broker where the subscription is done
> sends a not-so-short control-message to all other brokers. So far not too
> bad, but now comes the problem: Every broker seems to react on this
> control-message by sending another not-so-short control-message to all
> other
> brokers. This means, if N brokers are present, every subscription to a
> topic
> results in about (N-1)*N*2 control-messages (The additional factor two
> comes
> from the fact, that the final recipient-broker seems to send another
> response control-message). BTW, The length of the control-messages being
> sent seems to be closely related to the string length of the topic name.
>
> I have verified this behaviour with only up to 6 brokers, so i'm not
> totally
> shure if it will extrapolate to a larger number, but i suspect it does.
> This
> would mean, that with e.g. 100 Brokers every subscription to a topic would
> result in 100*99*2 control messages being sent. If a single control message
> has about 400 bytes (what i observed), this means a single subscription to
> a
> topic would result in about 8 Megabyte network traffic. This might even be
> sustainable, but the quadratic order kills scalabilty -- with 1000 Brokers
> we would have to transfer about 1 Gigabyte, what is no longer feasible.
>
> Finally, my questions: Is there any configuration option by which this
> quadratic order can be avoided, is my idea simply foolish, or are my
> conclusions wrong ? Note that i gained my conclusions more or less
> 'empirically' using wireshark.
>
> thanks for any advice, Martin
>
> --
> View this message in context:
> http://www.nabble.com/Scalability-of-%27Networks-of-Brokers%27-tp19335699p19335699.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Scalability of 'Networks of Brokers'
Posted by Martin_Cornelius <Ma...@smiths-heimann.com>.
The crucial problem is the requirement, that it shall possible to switch of
an arbitrary number of the machines. If i configure a cluster of standalone
brokers, and all of these brokers are accidentally switched of, the system
will stop to operate correctly, what is not permissible.
Joe Fernandez wrote:
>
> Instead of using such a potentially large network of 'embedded' brokers,
> could you use a smaller cluster of 'standalone' brokers? A single broker
> is capable of handling many clients and throwing in a master/slave
> configuration along with 'failover' connectors for the clients should
> provide the necessary high availability.
>
--
View this message in context: http://www.nabble.com/Scalability-of-%27Networks-of-Brokers%27-tp19335699p19366481.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Scalability of 'Networks of Brokers'
Posted by Joe Fernandez <jo...@ttmsolutions.com>.
Instead of using such a potentially large network of 'embedded' brokers,
could you use a smaller cluster of 'standalone' brokers? A single broker is
capable of handling many clients and throwing in a master/slave
configuration along with 'failover' connectors for the clients should
provide the necessary high availability.
Joe
http://www.ttmsolutions.com
Martin_Cornelius wrote:
>
> Hi all,
> <p>
> i am currently evaluating the scalability of Networks of ActiveMQ Brokers
> with a large number of brokers. The actual drive behind this evaluation: I
> have to design a distributed system that is completely serverless. It
> shall be possible to switch of any of the machines in the system without
> alloying the distributed functionality. The number of active machines in
> the system might vary from only 2 to about 100.<p>
> My current idea: Have an (presumably embedded) ActiveMQ Broker running on
> any of the machines, and let all this Brokers form a 'Network of Brokers'
> (abbreviated NOB in the follwing)
> <p>
> So far, i found that in an NOB a message is forwarded only to those
> brokers where at least one client has subscribed to the topic resp. queue
> the message is sent to. Really fine. If a client subscribes to a queue,
> the broker where the subscription is made sends a single short
> control-message to all other brokers of the NOB. That would still allow
> for thousands of brokers, fine again.
> <p>
> However, i just realized one fact that may compromise scalabilty: If a
> client subscribes to a topic, the broker where the subscription is done
> sends a not-so-short control-message to all other brokers. So far not too
> bad, but now comes the problem: Every broker seems to react on this
> control-message by sending another not-so-short control-message to all
> other brokers. This means, if N brokers are present, every subscription to
> a topic results in about (N-1)*N*2 control-messages (The additional factor
> two comes from the fact, that the final recipient-broker seems to send
> another response control-message). BTW, The length of the control-messages
> being sent seems to be closely related to the string length of the topic
> name.
> <p>
> I have verified this behaviour with only up to 6 brokers, so i'm not
> totally shure if it will extrapolate to a larger number, but i suspect it
> does. This would mean, that with e.g. 100 Brokers every subscription to a
> topic would result in 100*99*2 control messages being sent. If a single
> control message has about 400 bytes (what i observed), this means a single
> subscription to a topic would result in about 8 Megabyte network traffic.
> This might even be sustainable, but the quadratic order kills scalabilty
> -- with 1000 Brokers we would have to transfer about 1 Gigabyte, what is
> no longer feasible.
> <p>
> Finally, my questions: Is there any configuration option by which this
> quadratic order can be avoided, is my idea simply foolish, or are my
> conclusions wrong ? Note that i gained my conclusions more or less
> 'empirically' using wireshark.
> <p>
> thanks for any advice, Martin
>
>
--
View this message in context: http://www.nabble.com/Scalability-of-%27Networks-of-Brokers%27-tp19335699p19337053.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Scalability of 'Networks of Brokers'
Posted by Martin_Cornelius <Ma...@smiths-heimann.com>.
Gary Tully wrote:
>
> It may be worth while creating a jira for this so that it can be
> investigated a little further. This behavior will have an adverse
> effect on a network of any size. It should (I hope,) be possible to
> reduce the chattiness to some degree.
>
OK, JIRA entry is https://issues.apache.org/activemq/browse/AMQ-1932
AMQ-1932
--
View this message in context: http://www.nabble.com/Scalability-of-%27Networks-of-Brokers%27-tp19335699p19414798.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Scalability of 'Networks of Brokers'
Posted by Gary Tully <ga...@gmail.com>.
It may be worth while creating a jira for this so that it can be
investigated a little further. This behavior will have an adverse
effect on a network of any size. It should (I hope,) be possible to
reduce the chattiness to some degree.
2008/9/5 Martin_Cornelius <Ma...@smiths-heimann.com>:
>
> Hi all,
>
> i am currently evaluating the scalability of Networks of ActiveMQ Brokers
> with a large number of brokers. The actual drive behind this evaluation: I
> have to design a distributed system that is completely serverless. It shall
> be possible to switch of any of the machines in the system without alloying
> the distributed functionality. The number of active machines in the system
> might vary from only 2 to about 100.
>
> My current idea: Have an (presumably embedded) ActiveMQ Broker running on
> any of the machines, and let all this Brokers form a 'Network of Brokers'
> (abbreviated NOB in the follwing)
>
> So far, i found that in an NOB a message is forwarded only to those brokers
> where at least one client has subscribed to the topic resp. queue the
> message is sent to. Really fine. If a client subscribes to a queue, the
> broker where the subscription is made sends a single short control-message
> to all other brokers of the NOB. That would still allow for thousands of
> brokers, fine again.
>
> However, i just realized one fact that may compromise scalabilty: If a
> client subscribes to a topic, the broker where the subscription is done
> sends a not-so-short control-message to all other brokers. So far not too
> bad, but now comes the problem: Every broker seems to react on this
> control-message by sending another not-so-short control-message to all other
> brokers. This means, if N brokers are present, every subscription to a topic
> results in about (N-1)*N*2 control-messages (The additional factor two comes
> from the fact, that the final recipient-broker seems to send another
> response control-message). BTW, The length of the control-messages being
> sent seems to be closely related to the string length of the topic name.
>
> I have verified this behaviour with only up to 6 brokers, so i'm not totally
> shure if it will extrapolate to a larger number, but i suspect it does. This
> would mean, that with e.g. 100 Brokers every subscription to a topic would
> result in 100*99*2 control messages being sent. If a single control message
> has about 400 bytes (what i observed), this means a single subscription to a
> topic would result in about 8 Megabyte network traffic. This might even be
> sustainable, but the quadratic order kills scalabilty -- with 1000 Brokers
> we would have to transfer about 1 Gigabyte, what is no longer feasible.
>
> Finally, my questions: Is there any configuration option by which this
> quadratic order can be avoided, is my idea simply foolish, or are my
> conclusions wrong ? Note that i gained my conclusions more or less
> 'empirically' using wireshark.
>
> thanks for any advice, Martin
>
> --
> View this message in context: http://www.nabble.com/Scalability-of-%27Networks-of-Brokers%27-tp19335699p19335699.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>