How to reduce heap memory in artemis

[Sergey Fedechkin <se...@gmail.com>]

Hi. I asked this question here
https://stackoverflow.com/questions/79671868/heap-memory-use-more-than-global-max-size-specified,
but it seems like the topic couldn`t be bound just to one question, so here
I am.

I`ve done some load tests to check the heap consumption in broker paging
mode and found out, that heap could be reduced by limiting the parameters
follows:

   - global-max-size=200Mb
   - page-size-bytes=1M
   - max-read-page-bytes=2M
   - prefetch-page-messages=1 (that`s ok for me, because I`m not going to
   use multiple consumers of one queue)

Config with global-max-size=200Mb and other default values has the heap
around 1.4Gb.
Config with limited bytes (see above) has heap reduced to around 400Mb, and
that looks good.

Could you please confirm that I`m on the right path here? Are there any
other parameters critical to heap memory that I`ve missed?


Also in SO answer it was mentioned by Justin that the journal type won't
meaningfully change how much memory the broker uses.
But in tests I see that with same broker parameters (see above) for *large
messages* (280Kb) the NIO and JDBC journals show different heap memory
usage:

   - NIO - around 400Mb
   - JDBC - around 800Mb

Is it any JDBC-specific memory overhead that could cause higher heap usage?
--------------------------------

Producer client to load messages I`ve used:

> package org.acme.jms.producer;
>
> import java.nio.charset.StandardCharsets;
> import java.util.Set;
> import java.util.UUID;
> import java.util.concurrent.CountDownLatch;
> import java.util.concurrent.ExecutorService;
> import java.util.concurrent.Executors;
> import java.util.stream.Collectors;
> import java.util.stream.Stream;
>
> import javax.jms.BytesMessage;
> import javax.jms.DeliveryMode;
> import javax.jms.JMSContext;
> import javax.jms.JMSProducer;
> import javax.jms.Queue;
>
> import org.apache.qpid.jms.JmsConnectionFactory;
>
> public class MessageProducer
> {
>     static final int CLIENT_COUNT = 100;
>     static final int MESSAGE_COUNT = 200;
>     static final int PRODUCE_THREADS = 24;
>     static final String URL = "amqp://localhost:61616";
>     static final String USER = "usr";
>     static final String PWD = "pwd";
>     static final String MSG_70KB = Stream.generate(() -> "bytes test msg").limit(5000).collect(Collectors.joining());
>     static final String MSG_280KB = Stream.generate(() -> "bytes test msg").limit(20000).collect(Collectors.joining());
>
>     public static void main(String... args)
>     {
>         try
>         {
>             Set<String> queues = Stream.generate(() -> UUID.randomUUID().toString())
>                     .limit(CLIENT_COUNT)
>                     .collect(Collectors.toSet());
>
>             produceMessages(queues, MSG_70KB);
>             //produceMessages(queues, MSG_280KB);
>         }
>         catch (Exception e)
>         {
>             throw new RuntimeException(e);
>         }
>     }
>
>     private static void produceMessages(Set<String> queues, String msg) throws InterruptedException
>     {
>         CountDownLatch latch = new CountDownLatch(CLIENT_COUNT - 1);
>         ExecutorService executorService = Executors.newFixedThreadPool(PRODUCE_THREADS);
>
>         for (String queueName : queues)
>         {
>             executorService.execute(() ->
>             {
>                 JmsConnectionFactory jmsConnectionFactory = new JmsConnectionFactory(USER, PWD, URL);
>                 try (JMSContext jmsContext = jmsConnectionFactory.createContext(JMSContext.AUTO_ACKNOWLEDGE))
>                 {
>                     jmsContext.start();
>
>                     Queue queue = jmsContext.createQueue(queueName);
>                     JMSProducer producer = jmsContext.createProducer();
>                     producer.setDeliveryMode(DeliveryMode.PERSISTENT);
>
>                     for (int j = 0; j < MESSAGE_COUNT; j++)
>                     {
>                         BytesMessage jmsMsg = jmsContext.createBytesMessage();
>                         jmsMsg.setJMSDeliveryMode(DeliveryMode.PERSISTENT);
>                         jmsMsg.writeBytes(getTestMessage(msg).getBytes(StandardCharsets.UTF_8));
>
>                         producer.send(queue, jmsMsg);
>                     }
>                 }
>                 catch (Exception e)
>                 {
>                     throw new RuntimeException("jms msg send failed", e);
>                 }
>
>                 latch.countDown();
>             });
>         }
>         latch.await();
>         executorService.shutdown();
>     }
>
>     private static String getTestMessage(String msg)
>     {
>         return UUID.randomUUID() + msg;
>     }
> }
>
>

Regards, Sergey.

Re: How to reduce heap memory in artemis

[Clebert Suconic <cl...@apache.org>]

I would use prefetch as 10MB bytes... I wouldn't use a prefetch of one.. that would be too slow.


I would not also use global-max-size...  I want to change defaults on the broker on having every destination with a sensible max-size (such as 10MB).. if one destination falls behind you would have that destination flowing to disk only. 

Only reason I didn't do that is because this would represent a semanic change and I can only do that on an eventual Artemis 3.0... as some users use browser, filtering and they would prefer having all the messages in memory. (I argue about that though.. but that's where we are).


So.. have a max-size per destination at 10MB

Have you max-read at 10MB  and prefetch at 10MB


prefetch is just the blocks you use to read.. having it at one would make it calling too many tasks to depage data.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@activemq.apache.org
For additional commands, e-mail: users-help@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact