You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by ipolevoy <ig...@expresspigeon.com> on 2017/03/07 17:27:46 UTC
How does Apache Artemis manage heap space (RAM)
In our projects, we use two queues, one for normal processing, and another
for errors. The "error" queue sometimes gets filled with "error" messages,
which are sitting there, until a human checks them out. In some cases, the
"error" queue fills out with a lot of messages, and then the JVM runs out of
heap.
For instance, currently, the JVMs configured with max 4GB heap, and get OOM
exception at random times. We used a MemoryAnalyzer as well as IBM Heap
Analyzer and both point to Artemis. When I checked the size of the journal
on the file system, it is about 5GB.
We send:
* Small messages
* Persistent
* Not using Netty or remoting (just use Artemis for async processing)
* So, my questions are about Apache Artemis heap management and
recommendations:
1. Does it store persistent messages in RAM as well, even if they are
already stored on the file system?
2. If #1 is true, what is the strategy to control/limit amount od RAM
allocated by Artemis?
Also, this message is posted on SO:
http://stackoverflow.com/questions/42497606/how-does-apache-artemis-manage-heap-space-ram
Appreciate any help!
--
View this message in context: http://activemq.2283324.n4.nabble.com/How-does-Apache-Artemis-manage-heap-space-RAM-tp4723220.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: How does Apache Artemis manage heap space (RAM)
Posted by nigro_franz <ni...@gmail.com>.
You're welcome :)
I've to check how the DLQ queue's memory is managed, but I'm not expecting
it will have a life/weight on the HEAP besides just some metadata...
It is common to use it in that way, because if you're checking regularly the
messages, as Martyn pointed, the broker is pretty good to handle it anyway:
but is cool to
Yes the link is pretty useful!
--
View this message in context: http://activemq.2283324.n4.nabble.com/How-does-Apache-Artemis-manage-heap-space-RAM-tp4723220p4723450.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: [Apache Artemis][MQTT] how to create a durable subscriber - My
messages are lost
Posted by Deomisr <pa...@edf.fr>.
I solved my issue.
My issue came from the <expiry-delay>. his value was 300 i put 60000 and
all works fine.
--
View this message in context: http://activemq.2283324.n4.nabble.com/How-does-Apache-Artemis-manage-heap-space-RAM-tp4723220p4723534.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: [Apache Artemis][MQTT] how to create a durable subscriber - My
messages are lost
Posted by Deomisr <pa...@edf.fr>.
this is my consumer :
public class MQTTBasicSubExample {
public static void main(final String[] args) throws Exception {
// Create a new MQTT connection to the broker. We are not setting the
client ID. The broker will pick one for us.
System.out.println("Connecting to Artemis using MQTT");
MQTT mqtt = new MQTT();
mqtt.setConnectAttemptsMax(2);
mqtt.setReconnectAttemptsMax(1);
mqtt.setClientId("Test");
mqtt.setUserName("Admin");
mqtt.setPassword("manage");
BlockingConnection connection = null;
try{
connection = mqtt.blockingConnection();
connection.connect();
System.out.println("Connected to Artemis");
// Subscribe to topics
Topic[] topics = {new Topic("digital/test/data", QoS.EXACTLY_ONCE)};
connection.subscribe(topics);
System.out.println("Subscribed to topics.");
File f = new File("./lock/continue.lck");
while( !f.exists())
{
Message message = connection.receive(5, TimeUnit.SECONDS);
if(message != null)
{
System.out.println("Received messages.");
System.out.println(new String(message.getPayload()));
message.ack();
}
}
}
catch(Exception e)
{
e.printStackTrace();
}
finally{
System.out.println("deconnection");
if(connection!=null) connection.disconnect();
}
}
}
--
View this message in context: http://activemq.2283324.n4.nabble.com/How-does-Apache-Artemis-manage-heap-space-RAM-tp4723220p4723465.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: [Apache Artemis][MQTT] how to create a durable subscriber - My
messages are lost
Posted by Martyn Taylor <mt...@redhat.com>.
And what about on your consumer?
On Thu, Mar 9, 2017 at 3:43 PM, Deomisr <pa...@edf.fr> wrote:
> hi Mtaylor,
>
> I already put the mqtt.setclientid on my publisher
>
> this is my code :
>
> public class MQTTBasicPubSubExample {
>
> public static void main(final String[] args) throws Exception {
> // Create a new MQTT connection to the broker. We are not setting
> the
> client ID. The broker will pick one for us.
> System.out.println("Connecting to Artemis using MQTT");
> MQTT mqtt = new MQTT();
> mqtt.setConnectAttemptsMax(2);
> mqtt.setReconnectAttemptsMax(1);
> mqtt.setClientId("Test");
>
> mqtt.setUserName("CommerceDN");
> mqtt.setPassword("manage");
>
>
> mqtt.setHost("ssl://myServer:1883");
>
> BlockingConnection connection = mqtt.blockingConnection();
> connection.connect();
> System.out.println("Connected to Artemis");
>
> String payload = "This is message";
> connection.publish("digital/test/data", payload.getBytes(),
> QoS.EXACTLY_ONCE, false);
> System.out.println("Sent messages.");
>
> connection.disconnect();
> }
>
> }
>
>
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/How-does-Apache-Artemis-manage-heap-space-RAM-
> tp4723220p4723460.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: [Apache Artemis][MQTT] how to create a durable subscriber - My
messages are lost
Posted by Deomisr <pa...@edf.fr>.
hi Mtaylor,
I already put the mqtt.setclientid on my publisher
this is my code :
public class MQTTBasicPubSubExample {
public static void main(final String[] args) throws Exception {
// Create a new MQTT connection to the broker. We are not setting the
client ID. The broker will pick one for us.
System.out.println("Connecting to Artemis using MQTT");
MQTT mqtt = new MQTT();
mqtt.setConnectAttemptsMax(2);
mqtt.setReconnectAttemptsMax(1);
mqtt.setClientId("Test");
mqtt.setUserName("CommerceDN");
mqtt.setPassword("manage");
mqtt.setHost("ssl://myServer:1883");
BlockingConnection connection = mqtt.blockingConnection();
connection.connect();
System.out.println("Connected to Artemis");
String payload = "This is message";
connection.publish("digital/test/data", payload.getBytes(),
QoS.EXACTLY_ONCE, false);
System.out.println("Sent messages.");
connection.disconnect();
}
}
--
View this message in context: http://activemq.2283324.n4.nabble.com/How-does-Apache-Artemis-manage-heap-space-RAM-tp4723220p4723460.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: [Apache Artemis][MQTT] how to create a durable subscriber - My
messages are lost
Posted by Martyn Taylor <mt...@redhat.com>.
I suspect that you're not setting the client id.
mqtt.setClientId("my-unique-client");
On Thu, Mar 9, 2017 at 2:20 PM, REGINA Patrick <pa...@edf.fr>
wrote:
> Hi,
>
> How can I create a durable subscriber.
>
> Indeed when my consumer is disconnected, I lose my messages.
>
> I create a durable queue but nothing change.
> Broker.xml :
> ....
> <queues>
> <queue name="digital.test.data">
> <durable>true</durable>
> </queue>
> </queues>
> ......
>
This is not necessary, Artemis will create subscription queues for you.
>
> If I disconnect my consumer, when I reconnect it, I don't receive any
> messages.
>
> Exemple of subscriber :
> BlockingConnection connection = null;
> try{
> connection = mqtt.blockingConnection();
> connection.connect();
>
> Topic[] topics = {new Topic("digital/test/data",
> QoS.EXACTLY_ONCE)};
> connection.subscribe(topics);
>
> System.out.println("Subscribed to topics.");
>
> while( continue)
> {
> Message message = connection.receive(5, TimeUnit.SECONDS);
> if(message != null)
> {
> System.out.println("Received messages.");
> System.out.println(new String(message.getPayload()));
> message.ack();
> }
>
> }
> }
> catch(Exception e)
> {
> e.printStackTrace();
> }
> finally{
> if(connection!=null) connection.disconnect();
> }
>
> What i need to do in the broker.xml file.
>
> Best regards.
>
>
> Ce message et toutes les pièces jointes (ci-après le 'Message') sont
> établis à l'intention exclusive des destinataires et les informations qui y
> figurent sont strictement confidentielles. Toute utilisation de ce Message
> non conforme à sa destination, toute diffusion ou toute publication totale
> ou partielle, est interdite sauf autorisation expresse.
>
> Si vous n'êtes pas le destinataire de ce Message, il vous est interdit de
> le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou
> partie. Si vous avez reçu ce Message par erreur, merci de le supprimer de
> votre système, ainsi que toutes ses copies, et de n'en garder aucune trace
> sur quelque support que ce soit. Nous vous remercions également d'en
> avertir immédiatement l'expéditeur par retour du message.
>
> Il est impossible de garantir que les communications par messagerie
> électronique arrivent en temps utile, sont sécurisées ou dénuées de toute
> erreur ou virus.
> ____________________________________________________
>
> This message and any attachments (the 'Message') are intended solely for
> the addressees. The information contained in this Message is confidential.
> Any use of information contained in this Message not in accord with its
> purpose, any dissemination or disclosure, either whole or partial, is
> prohibited except formal approval.
>
> If you are not the addressee, you may not copy, forward, disclose or use
> any part of it. If you have received this message in error, please delete
> it and all copies from your system and notify the sender immediately by
> return message.
>
> E-mail communication cannot be guaranteed to be timely secure, error or
> virus-free.
>
>
[Apache Artemis][MQTT] how to create a durable subscriber - My
messages are lost
Posted by REGINA Patrick <pa...@edf.fr>.
Hi,
How can I create a durable subscriber.
Indeed when my consumer is disconnected, I lose my messages.
I create a durable queue but nothing change.
Broker.xml :
....
<queues>
<queue name="digital.test.data">
<durable>true</durable>
</queue>
</queues>
......
If I disconnect my consumer, when I reconnect it, I don't receive any messages.
Exemple of subscriber :
BlockingConnection connection = null;
try{
connection = mqtt.blockingConnection();
connection.connect();
Topic[] topics = {new Topic("digital/test/data", QoS.EXACTLY_ONCE)};
connection.subscribe(topics);
System.out.println("Subscribed to topics.");
while( continue)
{
Message message = connection.receive(5, TimeUnit.SECONDS);
if(message != null)
{
System.out.println("Received messages.");
System.out.println(new String(message.getPayload()));
message.ack();
}
}
}
catch(Exception e)
{
e.printStackTrace();
}
finally{
if(connection!=null) connection.disconnect();
}
What i need to do in the broker.xml file.
Best regards.
Re: How does Apache Artemis manage heap space (RAM)
Posted by ipolevoy <ig...@expresspigeon.com>.
haha, you are correct! Not only I should not use broker for error messages,
but having them in the DB is much more convenient. I could re-queue them, or
just delete. Thanks for the link to the antipattern!
--
View this message in context: http://activemq.2283324.n4.nabble.com/How-does-Apache-Artemis-manage-heap-space-RAM-tp4723220p4723416.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: How does Apache Artemis manage heap space (RAM)
Posted by nigro_franz <ni...@gmail.com>.
Hi!!!
AFAIK the en-queued persistent messages are in the journal and in the heap
too until consumed, but if Artemis should run low on memory it could page
them on disk (if a paging policy is configured).
Just to ask: why let the broker seems to behave like a database for the
error messages?
IMHO an efficient use of the broker is to let it be like a dispatcher/router
of messages and not as a final destination for them.
Consuming/moving them and using a proper tool (time series database or
others) to collect/query the error messages or rise alarm of any kind would
be more sympathetic with the broker mechanics and purpose.
Or I've misunderstood the way you're using the broker for the error queue?
P.S: there is a nice article
http://sensatic.net/messaging/messaging-anti-patterns-part-1.html
<http://sensatic.net/messaging/messaging-anti-patterns-part-1.html> on it
too!
--
View this message in context: http://activemq.2283324.n4.nabble.com/How-does-Apache-Artemis-manage-heap-space-RAM-tp4723220p4723308.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: How does Apache Artemis manage heap space (RAM)
Posted by ipolevoy <ig...@expresspigeon.com>.
Thanks for the info!
This makes sense. I explored the configuration in the debugger, and it does
not seem that Artemis may allocate more than 200MB of heap, but keeping all
these error messages in the queue might not be the best idea in the first
place.
Will Artemis use heap for DLQ?
<http://activemq.2283324.n4.nabble.com/file/n4723415/Selection_833.png>
--
View this message in context: http://activemq.2283324.n4.nabble.com/How-does-Apache-Artemis-manage-heap-space-RAM-tp4723220p4723415.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: How does Apache Artemis manage heap space (RAM)
Posted by Justin Bertram <jb...@apache.org>.
For what it's worth, the addressing model is actually going to change a fair bit in 2.0.0. There will be documentation to explain all that upon release so I won't go into the details here.
Justin
----- Original Message -----
From: "Justin Bertram" <jb...@apache.org>
To: users@activemq.apache.org
Sent: Thursday, March 9, 2017 9:59:58 AM
Subject: Re: How does Apache Artemis manage heap space (RAM)
I'd suggest reading these two bits of documentation:
http://activemq.apache.org/artemis/docs/1.5.3/using-core.html
http://activemq.apache.org/artemis/docs/1.5.3/jms-core-mapping.html
Justin
----- Original Message -----
From: "ipolevoy" <ig...@expresspigeon.com>
To: users@activemq.apache.org
Sent: Thursday, March 9, 2017 9:36:11 AM
Subject: Re: How does Apache Artemis manage heap space (RAM)
What is an address and how it relates to queues?
I see paging configuration the broker for "jms.queue.*"
I saw references to address in the docs here:
https://activemq.apache.org/artemis/docs/1.2.0/paging.html but still not
clear on what an address is.
--
View this message in context: http://activemq.2283324.n4.nabble.com/How-does-Apache-Artemis-manage-heap-space-RAM-tp4723220p4723457.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: How does Apache Artemis manage heap space (RAM)
Posted by Justin Bertram <jb...@apache.org>.
I'd suggest reading these two bits of documentation:
http://activemq.apache.org/artemis/docs/1.5.3/using-core.html
http://activemq.apache.org/artemis/docs/1.5.3/jms-core-mapping.html
Justin
----- Original Message -----
From: "ipolevoy" <ig...@expresspigeon.com>
To: users@activemq.apache.org
Sent: Thursday, March 9, 2017 9:36:11 AM
Subject: Re: How does Apache Artemis manage heap space (RAM)
What is an address and how it relates to queues?
I see paging configuration the broker for "jms.queue.*"
I saw references to address in the docs here:
https://activemq.apache.org/artemis/docs/1.2.0/paging.html but still not
clear on what an address is.
--
View this message in context: http://activemq.2283324.n4.nabble.com/How-does-Apache-Artemis-manage-heap-space-RAM-tp4723220p4723457.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: How does Apache Artemis manage heap space (RAM)
Posted by ipolevoy <ig...@expresspigeon.com>.
What is an address and how it relates to queues?
I see paging configuration the broker for "jms.queue.*"
I saw references to address in the docs here:
https://activemq.apache.org/artemis/docs/1.2.0/paging.html but still not
clear on what an address is.
--
View this message in context: http://activemq.2283324.n4.nabble.com/How-does-Apache-Artemis-manage-heap-space-RAM-tp4723220p4723457.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: How does Apache Artemis manage heap space (RAM)
Posted by Martyn Taylor <mt...@redhat.com>.
On Tue, Mar 7, 2017 at 5:27 PM, ipolevoy <ig...@expresspigeon.com> wrote:
> In our projects, we use two queues, one for normal processing, and another
> for errors. The "error" queue sometimes gets filled with "error" messages,
> which are sitting there, until a human checks them out. In some cases, the
> "error" queue fills out with a lot of messages, and then the JVM runs out
> of
> heap.
> For instance, currently, the JVMs configured with max 4GB heap, and get OOM
> exception at random times. We used a MemoryAnalyzer as well as IBM Heap
> Analyzer and both point to Artemis. When I checked the size of the journal
> on the file system, it is about 5GB.
>
> We send:
>
> * Small messages
> * Persistent
> * Not using Netty or remoting (just use Artemis for async processing)
> * So, my questions are about Apache Artemis heap management and
> recommendations:
>
> 1. Does it store persistent messages in RAM as well, even if they are
> already stored on the file system?
>
There are 3 reasons Artemis persists messages.
1. In the Message journal, the message journal is used only to recover
application state if the server is shutdown.
2. Large Messages can be streamed directly to disk to avoid taking up lots
of memory
3. Paging when an address runs out of memory Artemis (if appropriately
configured) will stop storing messages in memory and instead start writing
them directly to disk.
N.B. If messages are written directly to disk when Artemis receives, and
*not* stored in memory, we'd need to keep reading them from disk when
delivering to consumers. We have some optimisations when doing paging by
using caches, but it is still much slower.
2. If #1 is true, what is the strategy to control/limit amount od RAM
> allocated by Artemis?
>
There are multiple strategies (address-full-policies) for protecting the
broker from OOM issues. Check out the address-full-policy attribute of
address-settings:
https://activemq.apache.org/artemis/docs/1.5.3/queue-attributes.html
From what you've written here, I'd suggest enabling PAGING on your error
queue: https://activemq.apache.org/artemis/docs/1.5.3/paging.html. This
will allow the queue to grow large, until a user batch processes the errors.
Also, this message is posted on SO:
> http://stackoverflow.com/questions/42497606/how-does-
> apache-artemis-manage-heap-space-ram
>
> Appreciate any help!
>
> Regards
Martyn
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/How-does-Apache-Artemis-manage-heap-space-RAM-tp4723220.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>