RE: [EXTERNAL] RE: Artemis JMS queue paging prevents JMS consumer to receive newly added message

[Youyu Shao <ys...@crd.com>]

Thanks you very much Dondorp. We will certainly look into your suggestion.

Youyu

-----Original Message-----
From: Dondorp, Erwin <er...@cgi.com.INVALID>
Sent: Friday, November 5, 2021 12:40 PM
To: users@activemq.apache.org
Subject: [EXTERNAL] RE: Artemis JMS queue paging prevents JMS consumer to receive newly added message

Youyo,

I think this is expected behaviour.
As long as there are messages in memory, the paging files are not swapped-in yet.
This means that you might not get the messages with the desired colour in time when messages of other colours are still in memory.

A common solution is to use a single address and subscription queues on that address per colour, using the appropriate "selector".
(provided you have independent consumers for each colour) Each queue will then be filled with messages of only one colour.
And when you clean-up by consuming/purging, it would not be blocked by messages of different colours, since other colours are in other queues.

>  1.  is there any configuration setting one can apply to alter this behavior?
No

>  2.  Or this is a known 'issue'. If it is, any plan to rectify in the near future?
Any change to this behaviour would make message searching a dramatic experience since the system would then have to inspect ever disk-page for the requested messages.
That would certainly kill the performance of the server; especially when multiple clients can do that regularly.
But the above suggestion usually helps.

e.

-----Oorspronkelijk bericht-----
Van: Youyu Shao <ys...@crd.com>
Verzonden: vrijdag 5 november 2021 17:21
Aan: users@activemq.apache.org
Onderwerp: Artemis JMS queue paging prevents JMS consumer to receive newly added message


EXTERNAL SENDER:   Do not click any links or open any attachments unless you trust the sender and know the content is safe.
EXPÉDITEUR EXTERNE:    Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe à moins qu’ils ne proviennent d’un expéditeur fiable, ou que vous ayez l'assurance que le contenu provient d'une source sûre.

Hi,


While investigating performance related issues, we discovered that once Artemis JMS queue starts paging, newly produced message are not visible (within 60 seconds receive timeout) until most of the messages in the front (of that queue) are consumed.



We have Artemis (2.11.0) in production as JMS broker with 16G memory, where <max-size-bytes>-1</max-size-bytes> &  <address-full-policy>PAGE</address-full-policy>.



The following pseudo code is used to reproduce what we have experienced, in both Artemis (2.11.0) and Artemis (2.19.0):

  1.  The first parameter "conn" in those methods stands for the JMS connection.
  2.  The color parameter ('RED' and 'BLUE')  is used as message property.
  3.  The 'produce' method produces TextMessage with given color property.
  4.  The 'consume' method receives TextMessage of the given color property (using message SELECTOR), where receive timeout is 60 seconds
  5.  The 'clearMessage' clears the queue of messages of respective color





//1. clear all red messages

clearMessage(conn, queueName, "RED");



//2. clear all blue messages

clearMessage(conn, queueName, "BLUE");



//3. seeding the queue with 500_000 red messages, no paging yet

produce(conn, queueName, 500_000, "RED");



//4. produce 100 blue messages

produce(conn, queueName, 100, "BLUE") ;



//5. verify that one can successfully consume those 100 blue messages

consume(conn, queueName, 100, "BLUE");



//6. seeding another 9_000_000 red messages, to trigger paging

produce(conn, queueName, 9_000_000, "RED");



//7. produce 100 blue messages

produce(conn, queueName, 100, "BLUE");



//8. verify that one cannot receive those 100 blue messages within receive timeout 60 seconds

consume(conn, queueName, 100, "BLUE");



//9. consume 9_400_000 red messages to clear those in front of blue messages.

//   After this, there are 100_000 red messages in front of blue ones

consume(conn, queueName, 9_400_000, "RED");



//10. verify that it still cannot receive those 100 blue messages within receive timeout 60 seconds.

//     Even though this time there are only 100_00 red messages in front of blue ones

consume(conn, queueName, 100, "BLUE");



//11. consume anther 90_000 red messages to further clear those in front of blue messages,

//    After this, there are only 10_000 red messages in front of blue ones.

consume(conn, queueName, 90_000, "RED");



//12. yet, one still could not receive those 100 blue message within receive timeout 60 seconds.

//    This time there are only 10_000 red messages in front of them.

consume(conn, queueName, 100, "BLUE");



//13. consume anther 9_000 red messages to further clear those in front of blue messages,

//   After this, there are only 1_000 red messages in front of blue ones.

consume(conn, queueName, 9_000, "RED");



//14. finally, one can consumer those 100 blues messages, yes there are still 1_000 reds one in front them

consume(conn, queueName, 100, "BLUE");



Questions:

  1.  is there any configuration setting one can apply to alter this behavior?
  2.  Or this is a known 'issue'. If it is, any plan to rectify in the near future?



Thank you very much for your help,

Youyu




The information in this e-mail communication together with any attachments is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. If you are not the intended recipient of this e-mail communication, please notify us immediately. Any views expressed in this e-mail communication are those of the individual sender, unless otherwise specifically stated. Charles River Development does not represent, warrant or guarantee that the integrity of this communication has been maintained or that the communication is free of errors, virus or interference.


Information Classification *General
The information in this e-mail communication together with any attachments is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. If you are not the intended recipient of this e-mail communication, please notify us immediately. Any views expressed in this e-mail communication are those of the individual sender, unless otherwise specifically stated. Charles River Development does not represent, warrant or guarantee that the integrity of this communication has been maintained or that the communication is free of errors, virus or interference.

Information Classification *General