[jira] [Commented] (ARTEMIS-2404) Crash if the size of scheduled messages at a particular delivery time is greater than queue max size

["Justin Bertram (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ARTEMIS-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16873769#comment-16873769 ] 

Justin Bertram commented on ARTEMIS-2404:
-----------------------------------------

My first thought to mitigate an {{OutOfMemoryError}} would be to increase the amount of memory that the JVM has. You may simply not have enough memory to deal with your use-case. Or there could be a leak or a bug. To investigate further I think you'd need to reproduce on 2.9.0 and then attach a heap dump, logs, and broker.xml plus provide a more comprehensive description of the use-case (e.g. number of consumers, what the consumers are doing at the scheduled delivery time, how many messages are consumed before the OOME, etc.). It's worth noting that there is some additional memory overhead with scheduled messages. I'm not sure the efficiency of scheduled messages has ever been closely scrutinized for use at this kind of scale (i.e. millions of messages).

> Crash if the size of scheduled messages at a particular delivery time is greater than queue max size
> ----------------------------------------------------------------------------------------------------
>
>                 Key: ARTEMIS-2404
>                 URL: https://issues.apache.org/jira/browse/ARTEMIS-2404
>             Project: ActiveMQ Artemis
>          Issue Type: Bug
>    Affects Versions: 2.6.2
>            Reporter: Dave Myron
>            Priority: Critical
>
> In our use case, we need to schedule many messages for delivery at specific times of day (on occasion, 1-2 million messages will be scheduled for 8am the next day). We use paging. Our messages are 2-3KB in size. This leads to scheduled messages spread across several hundred page files (we use the default 10MB page size).
> At the scheduled delivery time, when getting the referenced messages, Artemis appears to naively attempt to read all the scheduled messages from the pages and, if there are enough messages, get itself into a situation where it runs out of memory.
> We're working on recreating the issue in a docker image (but will likely be unable to share it externally). We'll then be attempting to see if moving to 2.9.0 has any positive impact (though, looking at the code, it doesn't look like it will). In our usage, Artemis runs in a docker container and docker actually OOM killed the image (and then removed the image entirely, in its wisdom) – that was a very bad day. SMH…
> I see no other issues referencing this type of failure from depaging lots of scheduled deliveries. Are there suggestions on mitigating this kind of behavior?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)