You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@activemq.apache.org by Manuel Teira Paz <mt...@tid.es> on 2008/10/30 14:30:27 UTC
Database access and prefetch considerations
Hi, activemq guys.
We are suffering some bad service times on a production system embedding
ActiveMQ as message broker. We are not sure whether the problem is the
broker itself, but I would like to get some insights in the broker
internals, to be able to understand the situation better.
We are using an activemq 4.1 release (we are not allowed to upgrade it)
, with pure JDBC persistence. We are using only queues and persistent
messages.
The first concern is about database access. Even when the number of
messages in queues are high, we are observing no growth in the number of
active database connections. Furthermore, it seems that only a thread
per connection is responsible of fetching and deleting messages in the
persistence. Given the fact that we have 30 consumers on a given queue,
sharing the same connection, does this mean that a single thread is
managing all the persistence needs? If true, wouldn't it be a potential
bottleneck, avoiding us to improve the performance adding more consumers?
The second question is about the consumer prefetch. We did disable it in
the past, trying to avoid situations where messages got stuck in queues,
due to slow consumption of previous messages, producing unexpected and
apparently random delays in the consumption of some messages.
We wonder if prefetch could in some way lighten the load on the database
side. We guess that perhaps messages prefetched for a given queue are
kept in memory, avoiding the need to read them from the database. On the
other side, does prefetch activate a particular revover strategy for
database message restoring, or are they fetched in a one by one basis?
This also triggers the question of mesage caching. Even when prefetch is
disabled, is there any strategy, perhaps configurable, to cache messages
in memory? We are suspecting that when queue message counts goes higher,
performance goes down. Could this be related with cache issues, i.e. LRU
strategies keeping messages not to be delivered before older ones
already out of memory?
Please, don't hesitate into sharing any related theory with me. I hope
we can fix our performace problems with your help.
Best regards.
--
Manuel.
Re: Database access and prefetch considerations
Posted by Manuel Teira Paz <mt...@tid.es>.
Manuel Teira Paz escribió:
> Thanks a lot, James, for your fast answer. Taking advantage of your
> kindness, I will like to elaborate on some of the topics. Please, find
> my questions below:
>
Hi. Bumping this message, to see if anyone skilled in activemq insides
could answer my questions.
Best regards.
>
> James Strachan escribió:
>
>> 2008/10/30 Manuel Teira Paz <mt...@tid.es>:
>>
>>
>>> Hi, activemq guys.
>>>
>>>
>>> We are suffering some bad service times on a production system embedding
>>> ActiveMQ as message broker. We are not sure whether the problem is the
>>> broker itself, but I would like to get some insights in the broker
>>> internals, to be able to understand the situation better.
>>>
>>> We are using an activemq 4.1 release (we are not allowed to upgrade it) ,
>>> with pure JDBC persistence. We are using only queues and persistent
>>> messages.
>>>
>>> The first concern is about database access. Even when the number of messages
>>> in queues are high, we are observing no growth in the number of active
>>> database connections. Furthermore, it seems that only a thread per
>>> connection is responsible of fetching and deleting messages in the
>>> persistence.
>>>
>>>
>> Yes thats how it works.
>>
>>
>>
>>
>>> Given the fact that we have 30 consumers on a given queue,
>>> sharing the same connection, does this mean that a single thread is managing
>>> all the persistence needs? If true, wouldn't it be a potential bottleneck,
>>> avoiding us to improve the performance adding more consumers?
>>>
>>>
>> ActiveMQ is tuned for loads of concurrent connections. If you only
>> really have one connection, just don't share your connection across
>> each producer/consumer - or use async dispatch
>>
>>
>>
> Actually, we are not using a single connection for the whole system, but
> a single connection for every one of our components, handling message
> from a given queue. So, what is your recommendation, to use a single
> connection for every one of our 30 consumers, some ratio between
> consumers and connections?
>
> About the async dispatch possibility. What do actually involves,
> performance and reliably wise? I mean, how the database workload is
> delivered using async dispatch? Or, for example, should we be aware of
> asynchronous exceptions if we switch to that mode?
>
>>> The second question is about the consumer prefetch. We did disable it in the
>>> past, trying to avoid situations where messages got stuck in queues, due to
>>> slow consumption of previous messages, producing unexpected and apparently
>>> random delays in the consumption of some messages.
>>>
>>> We wonder if prefetch could in some way lighten the load on the database
>>> side. We guess that perhaps messages prefetched for a given queue are kept
>>> in memory, avoiding the need to read them from the database. On the other
>>> side, does prefetch activate a particular revover strategy for database
>>> message restoring, or are they fetched in a one by one basis?
>>>
>>>
>> They are fetched one by one - prefetch only affects how quickly they
>> are streamed to a consumer
>>
>>
>>
>>
>>> This also triggers the question of mesage caching. Even when prefetch is
>>> disabled, is there any strategy, perhaps configurable, to cache messages in
>>> memory?
>>>
>>>
>> Not in 4.x though there are improvements in 5.x
>>
>>
> Does that mean that there isn't any cache feature in 4.x or that it's
> not configurable?
> Could you please give more details about those improvements in 5.x?
>
>
>> BTW pure JDBC is always gonna be kinda slow - its why we developed AMQ
>> Store which is often 10X to 100X faster
>> http://activemq.apache.org/amq-message-store.html
>>
>>
>>
> The problem here is that we need the system to work in a master-slave
> configuration. I think that having some kind of short term fast
> persistence solution (as I guess amq store is) is not an option, since
> on a machine failure, database information is not guaranteed to be
> updated. Or am I missing something?
>
>
> Thanks a lot. Looking forward for your wise answers.
>
> Best regards.
>
>
>
>> --
>> James
>> -------
>> http://macstrac.blogspot.com/
>>
>> Open Source Integration
>> http://fusesource.com/
>>
>>
>
>
Re: Database access and prefetch considerations
Posted by Manuel Teira Paz <mt...@tid.es>.
Thanks a lot, James, for your fast answer. Taking advantage of your
kindness, I will like to elaborate on some of the topics. Please, find
my questions below:
James Strachan escribió:
> 2008/10/30 Manuel Teira Paz <mt...@tid.es>:
>
>> Hi, activemq guys.
>>
>>
>> We are suffering some bad service times on a production system embedding
>> ActiveMQ as message broker. We are not sure whether the problem is the
>> broker itself, but I would like to get some insights in the broker
>> internals, to be able to understand the situation better.
>>
>> We are using an activemq 4.1 release (we are not allowed to upgrade it) ,
>> with pure JDBC persistence. We are using only queues and persistent
>> messages.
>>
>> The first concern is about database access. Even when the number of messages
>> in queues are high, we are observing no growth in the number of active
>> database connections. Furthermore, it seems that only a thread per
>> connection is responsible of fetching and deleting messages in the
>> persistence.
>>
>
> Yes thats how it works.
>
>
>
>> Given the fact that we have 30 consumers on a given queue,
>> sharing the same connection, does this mean that a single thread is managing
>> all the persistence needs? If true, wouldn't it be a potential bottleneck,
>> avoiding us to improve the performance adding more consumers?
>>
>
> ActiveMQ is tuned for loads of concurrent connections. If you only
> really have one connection, just don't share your connection across
> each producer/consumer - or use async dispatch
>
>
Actually, we are not using a single connection for the whole system, but
a single connection for every one of our components, handling message
from a given queue. So, what is your recommendation, to use a single
connection for every one of our 30 consumers, some ratio between
consumers and connections?
About the async dispatch possibility. What do actually involves,
performance and reliably wise? I mean, how the database workload is
delivered using async dispatch? Or, for example, should we be aware of
asynchronous exceptions if we switch to that mode?
>
>> The second question is about the consumer prefetch. We did disable it in the
>> past, trying to avoid situations where messages got stuck in queues, due to
>> slow consumption of previous messages, producing unexpected and apparently
>> random delays in the consumption of some messages.
>>
>> We wonder if prefetch could in some way lighten the load on the database
>> side. We guess that perhaps messages prefetched for a given queue are kept
>> in memory, avoiding the need to read them from the database. On the other
>> side, does prefetch activate a particular revover strategy for database
>> message restoring, or are they fetched in a one by one basis?
>>
>
> They are fetched one by one - prefetch only affects how quickly they
> are streamed to a consumer
>
>
>
>> This also triggers the question of mesage caching. Even when prefetch is
>> disabled, is there any strategy, perhaps configurable, to cache messages in
>> memory?
>>
>
> Not in 4.x though there are improvements in 5.x
>
Does that mean that there isn't any cache feature in 4.x or that it's
not configurable?
Could you please give more details about those improvements in 5.x?
> BTW pure JDBC is always gonna be kinda slow - its why we developed AMQ
> Store which is often 10X to 100X faster
> http://activemq.apache.org/amq-message-store.html
>
>
The problem here is that we need the system to work in a master-slave
configuration. I think that having some kind of short term fast
persistence solution (as I guess amq store is) is not an option, since
on a machine failure, database information is not guaranteed to be
updated. Or am I missing something?
Thanks a lot. Looking forward for your wise answers.
Best regards.
> --
> James
> -------
> http://macstrac.blogspot.com/
>
> Open Source Integration
> http://fusesource.com/
>
Re: Database access and prefetch considerations
Posted by James Strachan <ja...@gmail.com>.
2008/10/30 Manuel Teira Paz <mt...@tid.es>:
> Hi, activemq guys.
>
>
> We are suffering some bad service times on a production system embedding
> ActiveMQ as message broker. We are not sure whether the problem is the
> broker itself, but I would like to get some insights in the broker
> internals, to be able to understand the situation better.
>
> We are using an activemq 4.1 release (we are not allowed to upgrade it) ,
> with pure JDBC persistence. We are using only queues and persistent
> messages.
>
> The first concern is about database access. Even when the number of messages
> in queues are high, we are observing no growth in the number of active
> database connections. Furthermore, it seems that only a thread per
> connection is responsible of fetching and deleting messages in the
> persistence.
Yes thats how it works.
> Given the fact that we have 30 consumers on a given queue,
> sharing the same connection, does this mean that a single thread is managing
> all the persistence needs? If true, wouldn't it be a potential bottleneck,
> avoiding us to improve the performance adding more consumers?
ActiveMQ is tuned for loads of concurrent connections. If you only
really have one connection, just don't share your connection across
each producer/consumer - or use async dispatch
> The second question is about the consumer prefetch. We did disable it in the
> past, trying to avoid situations where messages got stuck in queues, due to
> slow consumption of previous messages, producing unexpected and apparently
> random delays in the consumption of some messages.
>
> We wonder if prefetch could in some way lighten the load on the database
> side. We guess that perhaps messages prefetched for a given queue are kept
> in memory, avoiding the need to read them from the database. On the other
> side, does prefetch activate a particular revover strategy for database
> message restoring, or are they fetched in a one by one basis?
They are fetched one by one - prefetch only affects how quickly they
are streamed to a consumer
> This also triggers the question of mesage caching. Even when prefetch is
> disabled, is there any strategy, perhaps configurable, to cache messages in
> memory?
Not in 4.x though there are improvements in 5.x
BTW pure JDBC is always gonna be kinda slow - its why we developed AMQ
Store which is often 10X to 100X faster
http://activemq.apache.org/amq-message-store.html
--
James
-------
http://macstrac.blogspot.com/
Open Source Integration
http://fusesource.com/