You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Vjaceslavs Klimovs <vk...@gmail.com> on 2010/07/18 21:28:10 UTC
suitability of configuration for a use case
Hi,
We generally have two types of messages passing through our ActiveMQ
instances. (1) First type needs transacted guaranteed delivery and
high-availability, but the amount of messages of this kind is low, and
latency is not really important. (2) Second type, on the other hand,
needs low latency delivery, and there can be substantial amount of these
messages. However, reliability is not that important. (3) We would also
like to be able to take down brokers as needed, and of course we want
them to come up and resume operations when we are done with doing
whatever we were doing with the respective machine. (4) We don't have
SAN but we have clustered MySQL database. (5) We want to use JMX for
management.
This is where our findings start. Messages of type (1) should be
sent and received using JMS transactions, and should be persistent.
Messages of type (2) should be sent in auto acknowledgement mode and
should not be persistent. Because of HA requirement in (1) we need to
run two brokers. Because of (3) and (4) the only suitable
master/slave configuration is using JDBC. Because of the (2), small
limits should be placed on queues, and VM cursor should be used.
Now, how I think this is going to work. Any of the two brokers which
starts first, grabs a lock on the DB, becomes master and starts it's
transport connections. The other one that starts after, does not get a
lock and does not start transport connections. Clients, which are
configured to use failover:// transport, connect to either one and start
exchanging messages, either of type (1) or of type (2). Type (1)
messages are persisted to the database. If the master broker goes down,
slave gets the database lock, starts transport connectors and party
continues. If the one that went down comes up, it becomes slave and
waits for the lock. Type (2) messages are never persisted, and
therefore are very fast.
Based on all above, I've created configuration for brokers, it is here:
http://pastebin.com/YxV15k8M
I would be really grateful if someone could look through it, as well as
through my assumptions on the functioning of the system, and see if I
am wrong, missed something, or if something can be done better to
achieve stated requirements. I also have two concrete questions:
1) Is store usage counted when JDBC persistence is used?
2) VM cursor and small memory limits are used. Does that mean that
nothing will ever be paged out for non-persistent messages?
Sorry for the long post. Thank you in advance for any feedback.
/Vjaceslavs
Re: suitability of configuration for a use case
Posted by Dejan Bosanac <de...@nighttale.net>.
> 1) Is store usage counted when JDBC persistence is used?
store usage is not used for JDBC store
> 2) VM cursor and small memory limits are used. Does that mean that
> nothing will ever be paged out for non-persistent messages?
VM cursor will keep all you pending messages in memory. Small memory
limit (in combination with producer flow control) will adapt throttle
producers
Cheers
--
Dejan Bosanac - http://twitter.com/dejanb
Open Source Integration - http://fusesource.com/
ActiveMQ in Action - http://www.manning.com/snyder/
Blog - http://www.nighttale.net
> On Mon, 19 Jul 2010 10:00:48 +0200
> Dejan Bosanac <de...@nighttale.net> wrote:
>
>> Hi,
>>
>> at first glance I don't see anything wrong with your assumptions and
>> setup. You should try testing it in your environment and see if it
>> works as expected.
>>
>> Cheers
>> --
>> Dejan Bosanac - http://twitter.com/dejanb
>>
>> Open Source Integration - http://fusesource.com/
>> ActiveMQ in Action - http://www.manning.com/snyder/
>> Blog - http://www.nighttale.net
>>
>>
>>
>> On Sun, Jul 18, 2010 at 9:28 PM, Vjaceslavs Klimovs
>> <vk...@gmail.com> wrote:
>> > Hi,
>> >
>> > We generally have two types of messages passing through our ActiveMQ
>> > instances. (1) First type needs transacted guaranteed delivery and
>> > high-availability, but the amount of messages of this kind is low,
>> > and latency is not really important. (2) Second type, on the other
>> > hand, needs low latency delivery, and there can be substantial
>> > amount of these messages. However, reliability is not that
>> > important. (3) We would also like to be able to take down brokers
>> > as needed, and of course we want them to come up and resume
>> > operations when we are done with doing whatever we were doing with
>> > the respective machine. (4) We don't have SAN but we have clustered
>> > MySQL database. (5) We want to use JMX for management.
>> >
>> > This is where our findings start. Messages of type (1) should be
>> > sent and received using JMS transactions, and should be persistent.
>> > Messages of type (2) should be sent in auto acknowledgement mode and
>> > should not be persistent. Because of HA requirement in (1) we need
>> > to run two brokers. Because of (3) and (4) the only suitable
>> > master/slave configuration is using JDBC. Because of the (2), small
>> > limits should be placed on queues, and VM cursor should be used.
>> >
>> > Now, how I think this is going to work. Any of the two brokers which
>> > starts first, grabs a lock on the DB, becomes master and starts it's
>> > transport connections. The other one that starts after, does not
>> > get a lock and does not start transport connections. Clients, which
>> > are configured to use failover:// transport, connect to either one
>> > and start exchanging messages, either of type (1) or of type (2).
>> > Type (1) messages are persisted to the database. If the master
>> > broker goes down, slave gets the database lock, starts transport
>> > connectors and party continues. If the one that went down comes up,
>> > it becomes slave and waits for the lock. Type (2) messages are
>> > never persisted, and therefore are very fast.
>> >
>> > Based on all above, I've created configuration for brokers, it is
>> > here: http://pastebin.com/YxV15k8M
>> >
>> > I would be really grateful if someone could look through it, as
>> > well as through my assumptions on the functioning of the system,
>> > and see if I am wrong, missed something, or if something can be
>> > done better to achieve stated requirements. I also have two
>> > concrete questions: 1) Is store usage counted when JDBC persistence
>> > is used? 2) VM cursor and small memory limits are used. Does that
>> > mean that nothing will ever be paged out for non-persistent
>> > messages?
>> >
>> > Sorry for the long post. Thank you in advance for any feedback.
>> >
>> > /Vjaceslavs
>> >
>> >
>
>
Re: suitability of configuration for a use case
Posted by Vjaceslavs Klimovs <vk...@gmail.com>.
Thanks Dejan!
Two more questions:
1) Is store usage counted when JDBC persistence is used?
2) VM cursor and small memory limits are used. Does that mean that
nothing will ever be paged out for non-persistent messages?
On Mon, 19 Jul 2010 10:00:48 +0200
Dejan Bosanac <de...@nighttale.net> wrote:
> Hi,
>
> at first glance I don't see anything wrong with your assumptions and
> setup. You should try testing it in your environment and see if it
> works as expected.
>
> Cheers
> --
> Dejan Bosanac - http://twitter.com/dejanb
>
> Open Source Integration - http://fusesource.com/
> ActiveMQ in Action - http://www.manning.com/snyder/
> Blog - http://www.nighttale.net
>
>
>
> On Sun, Jul 18, 2010 at 9:28 PM, Vjaceslavs Klimovs
> <vk...@gmail.com> wrote:
> > Hi,
> >
> > We generally have two types of messages passing through our ActiveMQ
> > instances. (1) First type needs transacted guaranteed delivery and
> > high-availability, but the amount of messages of this kind is low,
> > and latency is not really important. (2) Second type, on the other
> > hand, needs low latency delivery, and there can be substantial
> > amount of these messages. However, reliability is not that
> > important. (3) We would also like to be able to take down brokers
> > as needed, and of course we want them to come up and resume
> > operations when we are done with doing whatever we were doing with
> > the respective machine. (4) We don't have SAN but we have clustered
> > MySQL database. (5) We want to use JMX for management.
> >
> > This is where our findings start. Messages of type (1) should be
> > sent and received using JMS transactions, and should be persistent.
> > Messages of type (2) should be sent in auto acknowledgement mode and
> > should not be persistent. Because of HA requirement in (1) we need
> > to run two brokers. Because of (3) and (4) the only suitable
> > master/slave configuration is using JDBC. Because of the (2), small
> > limits should be placed on queues, and VM cursor should be used.
> >
> > Now, how I think this is going to work. Any of the two brokers which
> > starts first, grabs a lock on the DB, becomes master and starts it's
> > transport connections. The other one that starts after, does not
> > get a lock and does not start transport connections. Clients, which
> > are configured to use failover:// transport, connect to either one
> > and start exchanging messages, either of type (1) or of type (2).
> > Type (1) messages are persisted to the database. If the master
> > broker goes down, slave gets the database lock, starts transport
> > connectors and party continues. If the one that went down comes up,
> > it becomes slave and waits for the lock. Type (2) messages are
> > never persisted, and therefore are very fast.
> >
> > Based on all above, I've created configuration for brokers, it is
> > here: http://pastebin.com/YxV15k8M
> >
> > I would be really grateful if someone could look through it, as
> > well as through my assumptions on the functioning of the system,
> > and see if I am wrong, missed something, or if something can be
> > done better to achieve stated requirements. I also have two
> > concrete questions: 1) Is store usage counted when JDBC persistence
> > is used? 2) VM cursor and small memory limits are used. Does that
> > mean that nothing will ever be paged out for non-persistent
> > messages?
> >
> > Sorry for the long post. Thank you in advance for any feedback.
> >
> > /Vjaceslavs
> >
> >
Re: suitability of configuration for a use case
Posted by Dejan Bosanac <de...@nighttale.net>.
Hi,
at first glance I don't see anything wrong with your assumptions and
setup. You should try testing it in your environment and see if it
works as expected.
Cheers
--
Dejan Bosanac - http://twitter.com/dejanb
Open Source Integration - http://fusesource.com/
ActiveMQ in Action - http://www.manning.com/snyder/
Blog - http://www.nighttale.net
On Sun, Jul 18, 2010 at 9:28 PM, Vjaceslavs Klimovs <vk...@gmail.com> wrote:
> Hi,
>
> We generally have two types of messages passing through our ActiveMQ
> instances. (1) First type needs transacted guaranteed delivery and
> high-availability, but the amount of messages of this kind is low, and
> latency is not really important. (2) Second type, on the other hand,
> needs low latency delivery, and there can be substantial amount of these
> messages. However, reliability is not that important. (3) We would also
> like to be able to take down brokers as needed, and of course we want
> them to come up and resume operations when we are done with doing
> whatever we were doing with the respective machine. (4) We don't have
> SAN but we have clustered MySQL database. (5) We want to use JMX for
> management.
>
> This is where our findings start. Messages of type (1) should be
> sent and received using JMS transactions, and should be persistent.
> Messages of type (2) should be sent in auto acknowledgement mode and
> should not be persistent. Because of HA requirement in (1) we need to
> run two brokers. Because of (3) and (4) the only suitable
> master/slave configuration is using JDBC. Because of the (2), small
> limits should be placed on queues, and VM cursor should be used.
>
> Now, how I think this is going to work. Any of the two brokers which
> starts first, grabs a lock on the DB, becomes master and starts it's
> transport connections. The other one that starts after, does not get a
> lock and does not start transport connections. Clients, which are
> configured to use failover:// transport, connect to either one and start
> exchanging messages, either of type (1) or of type (2). Type (1)
> messages are persisted to the database. If the master broker goes down,
> slave gets the database lock, starts transport connectors and party
> continues. If the one that went down comes up, it becomes slave and
> waits for the lock. Type (2) messages are never persisted, and
> therefore are very fast.
>
> Based on all above, I've created configuration for brokers, it is here:
> http://pastebin.com/YxV15k8M
>
> I would be really grateful if someone could look through it, as well as
> through my assumptions on the functioning of the system, and see if I
> am wrong, missed something, or if something can be done better to
> achieve stated requirements. I also have two concrete questions:
> 1) Is store usage counted when JDBC persistence is used?
> 2) VM cursor and small memory limits are used. Does that mean that
> nothing will ever be paged out for non-persistent messages?
>
> Sorry for the long post. Thank you in advance for any feedback.
>
> /Vjaceslavs
>
>