ActiveMQ persistence store is full. Stopping producer

[Kushal Gautam <ku...@gmail.com>]

Hi:

I have similar issues (
http://activemq.2283324.n4.nabble.com/Persistent-store-is-Full-100-of-107374182400-Stopping-producer-td4740024.html).
My activemq broker was mediating the transfer of millions of records, which
I was importing via Apache Camel (running on a
Karaf instance).

I am currently using ActiveMQ 5.15.4, and below is how my systemUsage
config
looks like:

<systemUsage>
    <systemUsage>
        <memoryUsage>
            <memoryUsage percentOfJvmHeap="70" />
        </memoryUsage>
        <storeUsage>
            <storeUsage limit="100 gb"/>
        </storeUsage>
        <tempUsage>
            <tempUsage limit="50 gb"/>
        </tempUsage>
    </systemUsage>
</systemUsage>

This is what I got in the logs:

Usage(default:store:queue://tst.tst.users:store) percentUsage=99%,
usage=42792917913, limit=40692963880,
percentUsageMinDelta=1%;Parent:Usage(default:store) percentUsage=105%,
usage=42792917913, limit=40692963880, percentUsageMinDelta=1%: Persistent
store is Full, 100% of 40692963880. Stopping producer
(ID:xxx-yy-0066-51437-1561640932972-1:24:3:1) to prevent flooding
queue://tst.tst.users. See
http://activemq.apache.org/producer-flow-control.html for more info
(blocking for: 228222s) | org.apache.activemq.broker.region.Queue |
ActiveMQ
Transport: tcp:///192.168.7.98:51461@61616

I have restarted the karaf instances, but this does not seem to take an
effect, and after looking at the broker logs, it quite made sense.

What can I actually do to avert this problem?

Currently, the broker stats for my queue shows:
Pending            No. of Consumers  Enqueued            Dequeued
8016495          1                          18094028           10077534

I am not sure, if restarting the broker will result in loss of these
messages.

Also, would be great if you could suggest about the config options that I
can make to optimize it better.

Regards,
Cooshal.

Re: ActiveMQ persistence store is full. Stopping producer

[Tim Bain <tb...@alumni.duke.edu>]

It appears that your consumer isn't keeping up with your producer(s),
filling the persistent store, which slows your producer(s) to the rate at
which the consumer is consuming.

Your options are:
* Do nothing, and let it complete at the current rate.
* Speed up consumption, either by adding consumers or making them faster or
both.
* Increase the limit for the persistent store to allow the producer to
write all remaining messages to the broker, so that the producer process
can exit even though the consumer is still processing messages. Note that
this requires a broker restart, so make sure your producer logic will
gracefully handle a period of broker unavailability. Persistent messages
will remain on the broker when a restart occurs, and since it's the
persistent store that is full, all of those messages will be kept, but if
you have any unconsumed non-persistent messages, they'll be lost.

Tim

On Mon, Jul 1, 2019, 5:51 AM Kushal Gautam <ku...@gmail.com> wrote:

> Hi:
>
> I have similar issues (
>
> http://activemq.2283324.n4.nabble.com/Persistent-store-is-Full-100-of-107374182400-Stopping-producer-td4740024.html
> ).
> My activemq broker was mediating the transfer of millions of records, which
> I was importing via Apache Camel (running on a
> Karaf instance).
>
> I am currently using ActiveMQ 5.15.4, and below is how my systemUsage
> config
> looks like:
>
> <systemUsage>
>     <systemUsage>
>         <memoryUsage>
>             <memoryUsage percentOfJvmHeap="70" />
>         </memoryUsage>
>         <storeUsage>
>             <storeUsage limit="100 gb"/>
>         </storeUsage>
>         <tempUsage>
>             <tempUsage limit="50 gb"/>
>         </tempUsage>
>     </systemUsage>
> </systemUsage>
>
> This is what I got in the logs:
>
> Usage(default:store:queue://tst.tst.users:store) percentUsage=99%,
> usage=42792917913, limit=40692963880,
> percentUsageMinDelta=1%;Parent:Usage(default:store) percentUsage=105%,
> usage=42792917913, limit=40692963880, percentUsageMinDelta=1%: Persistent
> store is Full, 100% of 40692963880. Stopping producer
> (ID:xxx-yy-0066-51437-1561640932972-1:24:3:1) to prevent flooding
> queue://tst.tst.users. See
> http://activemq.apache.org/producer-flow-control.html for more info
> (blocking for: 228222s) | org.apache.activemq.broker.region.Queue |
> ActiveMQ
> Transport: tcp:///192.168.7.98:51461@61616
>
> I have restarted the karaf instances, but this does not seem to take an
> effect, and after looking at the broker logs, it quite made sense.
>
> What can I actually do to avert this problem?
>
> Currently, the broker stats for my queue shows:
> Pending            No. of Consumers  Enqueued            Dequeued
> 8016495          1                          18094028           10077534
>
> I am not sure, if restarting the broker will result in loss of these
> messages.
>
> Also, would be great if you could suggest about the config options that I
> can make to optimize it better.
>
> Regards,
> Cooshal.
>