Questions to broker architecture and setup, memory configuration

[Tom_Z <to...@arcor.de>]

Hi there,

as an ActiveMQ starter I would like you to give me some feedback on my
setup.

Scenario is:
- currently single broker setup with standalone ActiveMQ 5.8 using LevelDB,
JRE 7, Protocol for all connected systems will be OpenWire
- 2 backend systems, let's call them B1 and B2, acting as producer and
consumer in sequential, non-multithreaded working mode
- B1 and B2 do produce different kinds of messages into their own individual
queue (two 'in'-queues from the client perspective)
- these systems can rely on a stable network connection and can be seen as a
fast producer and consumer running all over the day
- on the other side we do have a client application C with up to 250
instances working in parallel as a consumer on the two before mentioned
out-queues
- each client C does produce different kinds of response messages to
different 'out'-queues
- the clients do not have a stable network connection and messages can be
stuck in the queue for a while resulting in large out queues, at least quite
a bundle of messages for some of the consumers can remain for a longer time
- expected data volume: 50.000 messages a day, average size 10kb (range is
from 1kb to 1MB) 
- all messages must be persisted
- all queues and virtual destinations are predefined in activemq.xml

Message processing:
- B1 produces into QIB1 (in-queue for B1) and consumes from QO1, QO2, QO3
(queues-out 1 to 3), spread all over the day 
- B2 usually produces large amounts of messages only once a day into QIB2,
but consumes continiously returning messages from QO4
- every client C consumes request messages designated to that individual
client through JMSXGroupID from QIB1 and QIB2 and produces response messages
to QO1,2,3,4  
- in the opposite direction B1 and B2 are able to process messages for
theirselves delivered to their designated queues
- message order is relevant in both directions (in and out) 
- for testing purposes und to be able to redeliver messages to clients in
case of loss at client side we have defined "Composite Destinations", so
that each message is copied (forwardOnly="false") into an in- respectively
out-archive queue, so that messages can be pushed back into the relevant
queues (we are a little concerned about the quality of implementation on the
client side and finding bugs processing messages in production and like to
be prepared for such cases, archiving will be deactivated later)

1. Is that a suitable setup to use ActiveMQ in production?

2. Is a composite destination from queue to queue okay? In the documentation
often topics are used, from the archive queues will never be consumed - only
copied, deleted or purged and viewed in the queue browser.

3. Is it possible to add an expiry date while forwarding to the archive
queue/topic (Apache Camel is not in use)?

4. One open question is whether to use a single out queue (easier for client
app C) and to use filtered destinations, filtering must then base on a
application specific message property to distribute the message types to the
different out-queues, is that possible? Does it perhaps perform better, on
client and server side?

5. I still have to take a deeper look at the memory settings. The queues do
have quite different characteristics, from being constantly emptied (out
queues) to holding large amounts of messages for lots of clients for some
time being able to deliver at any time (in queues). I guess at least
regarding to both out queues containing messages for up to 250 different
message groups I have to do some finetuning? 

6. What are the side effects of too little memory for queue? Out-queues are
not time or performance critical and B1 and B2 should grab all messages
regardless of the message group, in-queues must deliver fast to the client
according to the requested message group. 

I found a note: "like message selector matching, grouping based on
JMSXGroupID occurs before dispatch on messages in memory. With the default
maxPageSize setting, large backlogs of messages destined for one group they
can block receipt of messages to other groups if they don't all fit in
memory." 

That means if memory is not enough for all(!) groups/messages in the
in-queue, a client possibly receives only a subset of messages for his
message group?

Thanks for any hints and advice,
Tom



--
View this message in context: http://activemq.2283324.n4.nabble.com/Questions-to-broker-architecture-and-setup-memory-configuration-tp4679749.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Questions to broker architecture and setup, memory configuration

[Tom_Z <to...@arcor.de>]

ceposta wrote
> So this a little TL;DR... but I try answer some of your questions
> inline...

Hi Christian, sorry, what does that mean? :-) TL, DR?


ceposta wrote
> If I read correctly, you have only one instance running? No.. this is
> not a suitable setup for production. What happens if/when that one
> instance fails? You should think about having at least a Slave broker
> in warm standy.

Sorry, that was misleading. I only wanted to express not to use a network of
brokers. In production environment we will add a slave instance.


ceposta wrote
> The messages will never be consumed?
> This will not work for long periods, you will run out of disk space.
> Why not just store to a separate, longer-term storage? Sounds like
> you're trying to use ActiveMQ for both a real-time even broker and
> some kind of longer-term audit log... not a good idea.

I'm aware that this is not in sense of messaging but as I stated this will
be an temporary feature with purging in between. We do not want to establish
more infrastructure beside the broker and like to be able to easily push
back to queues. What long term alternative is able to do so? As questioned
in no 3, ideally we would add an expiry date, so that archive is not growing
but has quite a constant number of messages until finally deactivated. We
can add this at client side as message property, but server side would be
preferable.


ceposta wrote
>>
>> 3. Is it possible to add an expiry date while forwarding to the archive
>> queue/topic (Apache Camel is not in use)?
>>
>> 4. One open question is whether to use a single out queue (easier for
>> client
>> app C) and to use filtered destinations, filtering must then base on a
>> application specific message property to distribute the message types to
>> the
>> different out-queues, is that possible? Does it perhaps perform better,
>> on
>> client and server side?
> 
> Depends on how many messages and how much you're filtering...If the
> producers know the partition ahead of time, then just stick with that
> until it doesn't scale/work.

That is not so easy to answer, but I try to do an estimated guess: 50.000
messages per day, let's 35.000 in the out-queues split QO1=20.000,
QO2=10.000, QO3=4.000, QO4=1.000. Would filtering in that rough case a
better way?


ceposta wrote
> A couple things... message groups are honored based on messages that
> have been paged into memory.
> 
> http://activemq.apache.org/message-groups.html
> 
> If you have too little memory allocated to broker or destinations,
> then messages will need to be paged in from disk (which can be
> slower).

That means for a requested message group some messages may be paged in
memory but other non-paged messages will be read from disk but all together
will be delivered to the client in the same connection or can only those
messages be delivered for a message group that are already paged in memory?

Thanks,
Tom





--
View this message in context: http://activemq.2283324.n4.nabble.com/Questions-to-broker-architecture-and-setup-memory-configuration-tp4679749p4679784.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Questions to broker architecture and setup, memory configuration

[Christian Posta <ch...@gmail.com>]

So this a little TL;DR... but I try answer some of your questions inline...

On Mon, Mar 31, 2014 at 11:53 AM, Tom_Z <to...@arcor.de> wrote:
> Hi there,
>
> as an ActiveMQ starter I would like you to give me some feedback on my
> setup.
>
> Scenario is:
> - currently single broker setup with standalone ActiveMQ 5.8 using LevelDB,
> JRE 7, Protocol for all connected systems will be OpenWire
> - 2 backend systems, let's call them B1 and B2, acting as producer and
> consumer in sequential, non-multithreaded working mode
> - B1 and B2 do produce different kinds of messages into their own individual
> queue (two 'in'-queues from the client perspective)
> - these systems can rely on a stable network connection and can be seen as a
> fast producer and consumer running all over the day
> - on the other side we do have a client application C with up to 250
> instances working in parallel as a consumer on the two before mentioned
> out-queues
> - each client C does produce different kinds of response messages to
> different 'out'-queues
> - the clients do not have a stable network connection and messages can be
> stuck in the queue for a while resulting in large out queues, at least quite
> a bundle of messages for some of the consumers can remain for a longer time
> - expected data volume: 50.000 messages a day, average size 10kb (range is
> from 1kb to 1MB)
> - all messages must be persisted
> - all queues and virtual destinations are predefined in activemq.xml
>
> Message processing:
> - B1 produces into QIB1 (in-queue for B1) and consumes from QO1, QO2, QO3
> (queues-out 1 to 3), spread all over the day
> - B2 usually produces large amounts of messages only once a day into QIB2,
> but consumes continiously returning messages from QO4
> - every client C consumes request messages designated to that individual
> client through JMSXGroupID from QIB1 and QIB2 and produces response messages
> to QO1,2,3,4
> - in the opposite direction B1 and B2 are able to process messages for
> theirselves delivered to their designated queues
> - message order is relevant in both directions (in and out)
> - for testing purposes und to be able to redeliver messages to clients in
> case of loss at client side we have defined "Composite Destinations", so
> that each message is copied (forwardOnly="false") into an in- respectively
> out-archive queue, so that messages can be pushed back into the relevant
> queues (we are a little concerned about the quality of implementation on the
> client side and finding bugs processing messages in production and like to
> be prepared for such cases, archiving will be deactivated later)
>
> 1. Is that a suitable setup to use ActiveMQ in production?

If I read correctly, you have only one instance running? No.. this is
not a suitable setup for production. What happens if/when that one
instance fails? You should think about having at least a Slave broker
in warm standy.

>
> 2. Is a composite destination from queue to queue okay? In the documentation
> often topics are used, from the archive queues will never be consumed - only
> copied, deleted or purged and viewed in the queue browser.

The messages will never be consumed?
This will not work for long periods, you will run out of disk space.
Why not just store to a separate, longer-term storage? Sounds like
you're trying to use ActiveMQ for both a real-time even broker and
some kind of longer-term audit log... not a good idea.

>
> 3. Is it possible to add an expiry date while forwarding to the archive
> queue/topic (Apache Camel is not in use)?
>
> 4. One open question is whether to use a single out queue (easier for client
> app C) and to use filtered destinations, filtering must then base on a
> application specific message property to distribute the message types to the
> different out-queues, is that possible? Does it perhaps perform better, on
> client and server side?

Depends on how many messages and how much you're filtering...If the
producers know the partition ahead of time, then just stick with that
until it doesn't scale/work.

>
> 5. I still have to take a deeper look at the memory settings. The queues do
> have quite different characteristics, from being constantly emptied (out
> queues) to holding large amounts of messages for lots of clients for some
> time being able to deliver at any time (in queues). I guess at least
> regarding to both out queues containing messages for up to 250 different
> message groups I have to do some finetuning?
>
> 6. What are the side effects of too little memory for queue?
A couple things... message groups are honored based on messages that
have been paged into memory.

http://activemq.apache.org/message-groups.html

If you have too little memory allocated to broker or destinations,
then messages will need to be paged in from disk (which can be
slower).


Out-queues are
> not time or performance critical and B1 and B2 should grab all messages
> regardless of the message group, in-queues must deliver fast to the client
> according to the requested message group.
>
> I found a note: "like message selector matching, grouping based on
> JMSXGroupID occurs before dispatch on messages in memory. With the default
> maxPageSize setting, large backlogs of messages destined for one group they
> can block receipt of messages to other groups if they don't all fit in
> memory."
>
> That means if memory is not enough for all(!) groups/messages in the
> in-queue, a client possibly receives only a subset of messages for his
> message group?
>
> Thanks for any hints and advice,
> Tom
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Questions-to-broker-architecture-and-setup-memory-configuration-tp4679749.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.



-- 
Christian Posta
http://www.christianposta.com/blog
twitter: @christianposta