Strange behaviour when broker ran out of disk space

[BlueFox <bl...@gmail.com>]

Hi, I'm encountering a strange behaviour on ActiveMQ 5.1 broker using
AMQStore for persistence. In my setup, an ActiveMQCPP (2.1.3) producer with
AUTO_ACKNOWLEDGE is sending persistent message to a queue, which is being
processed slowly on a consumer using VMTransport on same VM as the broker. I
don't have any special setting for prefetch or memory/disk limit etc.

When the broker disk space ran out on the broker, the client would receive
out of disk space error, which is fine. Then the client would treat this as
a network failure and still keep try to send the message to the broker, this
repeats for a while as the broker memory usage goes up (!!!) until it
finally ran out of memory. My question is does it mean that the producer
need to treat each openwire error differently in order to avoid killing the
broker? In this case (the broker ran out of disk space), since the broker
cannot accept any more messages, shouldn't it simply drop any new messages
coming in until more disk space is available instead of keeping all the new
messages in the memory?
-- 
View this message in context: http://www.nabble.com/Strange-behaviour-when-broker-ran-out-of-disk-space-tp17843784p17843784.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Strange behaviour when broker ran out of disk space

[BlueFox <bl...@gmail.com>]

Found the second problem, it is in the DataFileAppender & AsyncDatamanager,
when the disk is full, the message is saved in the inflightWrites map but
never taken out since the enqueue will fail due to IOException.

        synchronized (this) {
            // Find the position where this item will land at.
            DataFile dataFile = dataManager.allocateLocation(location);
            if( !sync ) {
                inflightWrites.put(new WriteKey(location), write);
            }
            batch = enqueue(dataFile, write);
        }

and

        synchronized (this) {
            // Find the position where this item will land at.
            DataFile dataFile = dataManager.allocateLocation(location);
            inflightWrites.put(new WriteKey(location), write);
            batch = enqueue(dataFile, write);
        }

>From the above 5.1 source code, it seems like allocateLocation should
actually make sure that there is enough disk space for the particular
message. However, it actually only create the File handle without actually
allocating any disk space So to fix this problem, we added 

nextWriteFile.closeRandomAccessFile(nextWriteFile.openRandomAccessFile(true));

in the AsyncDataManager.allocateLocation method to try and pre allocate the
disk space. We believe that this will not have much impact on the
performance as data file is 32MB default, but please feel free to advice
otherwise as this is our first time looking at ActiveMQ source code.

Regards,


BlueFox wrote:
> 
> 
> 
> BlueFox wrote:
>> 
>> Hi, I'm encountering a strange behaviour on ActiveMQ 5.1 broker using
>> AMQStore for persistence. In my setup, an ActiveMQCPP (2.1.3) producer
>> with AUTO_ACKNOWLEDGE is sending persistent message to a queue, which is
>> being processed slowly on a consumer using VMTransport on same VM as the
>> broker. I don't have any special setting for prefetch or memory/disk
>> limit etc.
>> 
>> When the broker disk space ran out on the broker, the client would
>> receive out of disk space error, which is fine. Then the client would
>> treat this as a network failure and still keep try to send the message to
>> the broker, this repeats for a while as the broker memory usage goes up
>> (!!!) until it finally ran out of memory. My question is does it mean
>> that the producer need to treat each openwire error differently in order
>> to avoid killing the broker? In this case (the broker ran out of disk
>> space), since the broker cannot accept any more messages, shouldn't it
>> simply drop any new messages coming in until more disk space is available
>> instead of keeping all the new messages in the memory?
>> 
> 
> I found the problem is a part of issue listed in
> https://issues.apache.org/activemq/browse/AMQ-724. However, I now
> encounter another problem where the persistent message is left in the
> memory, even though the disk error should reject the message....
> 

-- 
View this message in context: http://www.nabble.com/Strange-behaviour-when-broker-ran-out-of-disk-space-tp17843784p17909860.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Strange behaviour when broker ran out of disk space

[BlueFox <bl...@gmail.com>]

BlueFox wrote:
> 
> Hi, I'm encountering a strange behaviour on ActiveMQ 5.1 broker using
> AMQStore for persistence. In my setup, an ActiveMQCPP (2.1.3) producer
> with AUTO_ACKNOWLEDGE is sending persistent message to a queue, which is
> being processed slowly on a consumer using VMTransport on same VM as the
> broker. I don't have any special setting for prefetch or memory/disk limit
> etc.
> 
> When the broker disk space ran out on the broker, the client would receive
> out of disk space error, which is fine. Then the client would treat this
> as a network failure and still keep try to send the message to the broker,
> this repeats for a while as the broker memory usage goes up (!!!) until it
> finally ran out of memory. My question is does it mean that the producer
> need to treat each openwire error differently in order to avoid killing
> the broker? In this case (the broker ran out of disk space), since the
> broker cannot accept any more messages, shouldn't it simply drop any new
> messages coming in until more disk space is available instead of keeping
> all the new messages in the memory?
> 

I found the problem is a part of issue listed in
https://issues.apache.org/activemq/browse/AMQ-724. However, I now encounter
another problem where the persistent message is left in the memory, even
though the disk error should reject the message....
-- 
View this message in context: http://www.nabble.com/Strange-behaviour-when-broker-ran-out-of-disk-space-tp17843784p17869391.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.