Paging performance issue

[Sergey Zagalskiy <se...@mail.ru>]

Hi, Artemis team!
I have some problem with Artemis paging in my highload application.
In a nutshell, in my application producers produces messages very often -
about 500 messages per second.
At the same time, consumer is sometimes pretty slow. So consumer processess
messages slower than producer produces them. In this case why messages
accumulate in queue, clog it up and application fails with OutOfMemoryError.
I'm trying to use paging for "mitigating" this problem. As a result,
application doesn't fail with OutOfMemoryError, messages are stored on disk
- not in memory.
But... I have an issue here. If application paged a lot of messages (100k)
once - consumer can not process messages from paging and "new" messages from
queue. Even if consumer in very fast - even if he doesn't do anything during
processing.
So, if application jumped to paging once - it will never recover its initial
message processing speed.
 
This is how I test it on my local environment:
I have following jms architecture

Producer => QueueOUT => Bridge => QueueIN => Consumer.

Consumer DOES NOT DO ANYTHING. This is just a listener for queue with blank
method.

I stop the bridge.
Wait until  QueueOUT will have 100k messages
Start a bridge - messages are transfered to QueueIN and Consumers starts to
process them.
At the same time producer produces new messages, which are transfered to
QueueIN. But consumer can not proccess "old" messages form paging and "new"
messages, which just delivered from producer.

As I understood from investigation of the paging code - method
org.apache.activemq.artemis.core.paging.impl.Page.read(StorageManager)
Artemis gets messages from disk one by one this algorithm is hardcoded. And
there are no settings to be configured for Artemis batch retrieving messages
from disk.

Could you please, clarify me does paging can be tuned for bath-retrievig
messages form disk for the faster message processing.
And, in general, is paging a good solution for my case, may be there are
some better solutions for solving my problem?

Thanks in advance for your help 
Sergey




--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-Dev-f2368404.html

Re: Paging performance issue

[Justin Bertram <jb...@apache.org>]

> Could you please, clarify me does paging can be tuned for bath-retrievig
messages form disk for the faster message processing.

The method you referenced as getting "messages from disk one by one"
(i.e. org.apache.activemq.artemis.core.paging.impl.Page.read(StorageManager))
isn't doing quite what you describe.  It isn't reading a single
_message_ from disk; it's reading an entire _page_ from disk.  Each page
consists of multiple messages (configurable via page-size-bytes).  Take a
look at the paging documentation [1] for more details.


> And, in general, is paging a good solution for my case, may be there are
some better solutions for solving my problem?

You might be better served by using flow-control to throttle the producers
and/or using the BLOCK address-full-policy.

At the end of the day if you produce messages faster than you consume them
you're going to eventually run out of resources on the broker.  You'll hit
your memory limit first, and then (if paging is configured) eventually
you'll hit your disk limit.  The only real way to recover from such a
situation is to slow down production or increase consumption such that the
messages consumed "catches up" to the messages that have been produced.
You can increase performance by giving the broker more memory (i.e. larger
address-full-size), but anytime you start paging you'll be working with a
_much_ slower medium than RAM (whether that's spinning HDDs or SSDs).


Justin

[1] http://activemq.apache.org/artemis/docs/latest/paging.html

On Sun, Oct 1, 2017 at 5:07 AM, Sergey Zagalskiy <se...@mail.ru>
wrote:

> Hi, Artemis team!
> I have some problem with Artemis paging in my highload application.
> In a nutshell, in my application producers produces messages very often -
> about 500 messages per second.
> At the same time, consumer is sometimes pretty slow. So consumer processess
> messages slower than producer produces them. In this case why messages
> accumulate in queue, clog it up and application fails with
> OutOfMemoryError.
> I'm trying to use paging for "mitigating" this problem. As a result,
> application doesn't fail with OutOfMemoryError, messages are stored on disk
> - not in memory.
> But... I have an issue here. If application paged a lot of messages (100k)
> once - consumer can not process messages from paging and "new" messages
> from
> queue. Even if consumer in very fast - even if he doesn't do anything
> during
> processing.
> So, if application jumped to paging once - it will never recover its
> initial
> message processing speed.
>
> This is how I test it on my local environment:
> I have following jms architecture
>
> Producer => QueueOUT => Bridge => QueueIN => Consumer.
>
> Consumer DOES NOT DO ANYTHING. This is just a listener for queue with blank
> method.
>
> I stop the bridge.
> Wait until  QueueOUT will have 100k messages
> Start a bridge - messages are transfered to QueueIN and Consumers starts to
> process them.
> At the same time producer produces new messages, which are transfered to
> QueueIN. But consumer can not proccess "old" messages form paging and "new"
> messages, which just delivered from producer.
>
> As I understood from investigation of the paging code - method
> org.apache.activemq.artemis.core.paging.impl.Page.read(StorageManager)
> Artemis gets messages from disk one by one this algorithm is hardcoded. And
> there are no settings to be configured for Artemis batch retrieving
> messages
> from disk.
>
> Could you please, clarify me does paging can be tuned for bath-retrievig
> messages form disk for the faster message processing.
> And, in general, is paging a good solution for my case, may be there are
> some better solutions for solving my problem?
>
> Thanks in advance for your help
> Sergey
>
>
>
>
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-Dev-
> f2368404.html
>