Qpid-cpp-server-linearstore problem with growing up files infinitely.

[Adrian Cislo <Ad...@fara.no>]

Hello,

We have been recently suffering a quite critical problem with journal files on production environment.
After some time, even though queues seem to be empty, Qpid allocates a lot of disk space for journal files.

Qpid is in version 0.30, rhel6 distribution. All running on RHEL 6.5

Enclosed you will find some sample logs from that environment:

9.12.15_jrnl_size.txt - results from du command
Qpid-stat-q.txt - results from qpid-stat -q
qpid_vsi_qpid-tool-result.txt, qpid_request_qpid-tool-result.txt, qpid_public_qpid-tool-result.txt - results from qpid-tool, show #

The question is, why does it happen, and if/how we can prevent it ?


Best regards

FARA
Adrian Cis�o
Software Engineer
Mobile: +48790618366
E-mail : adrian.cislo@fara.no<ma...@fara.no>

TECHNOLOGY IN MOTION

Re: Qpid-cpp-server-linearstore problem with growing up files infinitely.

[Kim van der Riet <ki...@redhat.com>]

This problem can occur if the messages are consumed out-of-order, or 
more specifically, if there exists and "orphaned" message which has been 
enqueued some time back, but not consumed.

The linearstore assumes that all messages are consumed (more-or-less) in 
the order they are sent to the broker.  Files are freed from the 
linearstore as the "leading" files become freed of all stored messages. 
  If a message should be enqueued but not consumed, then the journal 
file containing that message will not be freed, and the files following 
that file will also be blocked from being freed (as this would 
compromise the ability to recover the state of the store if there should 
be a stoppage).

If your messaging scenario does not meet the in-order criterion, and 
message consumption is random with possible long periods of storage for 
some messages, then it is possible that linearstore will not be a 
suitable store.  A store with random read/write (possibly based on a 
database) would be better for this use-case.

I see from your logs that a single message is enqueued. If that message 
is an orphaned message then that would explain the growth in journal 
files. Assuming this to be true:

Strategies for preventing this depend on your use-case:

1. The best way is to avoid orphaned messages in the first place. This 
will ensure that the journal is always short and the individual journal 
files are being re-used.

2. If the presence of unconsumed messages for long periods is 
unavoidable and the orphaned messages are not wanted, then one way to 
control this is to set a Time-to-Live on each message such that when 
that expires, the message is deleted by the broker automatically, which 
would have the effect of freeing up the linearstore journal files. (I 
have not tested this strategy, but I believe it should work.)

3. The last and more complex way to handle this is to have a small 
client find and either remove the messages (if they are unwanted), or 
remove and resend the messages at regular intervals (if the messages are 
wanted and must be preserved).

If the message in your log is recent, and not orphaned, then there could 
be a bug in the store which needs addressing.  If this is so, then it 
would be helpful (if possible) to get a copy of the store files so that 
I can analyze why the process of freeing them is "stuck".

There is a proposal to add a limit to the queue expansion (ie set a 
maximum number of files allowed by a single queue and/or combined 
queues. After the limit is exceeded, further enqueues would be refused. 
I am not sure if such a feature would help your situation, as it would 
prevent unlimited growth in journal files, but at the cost of having 
client message publishing operations fail owing to the full store.

I hope this helps,
Kim van der Riet

On 12/09/2015 08:50 AM, Adrian Cislo wrote:
> Hello,
>
> We have been recently suffering a quite critical problem with journal
> files on production environment.
>
> After some time, even though queues seem to be empty, Qpid allocates a
> lot of disk space for journal files.
>
> Qpid is in version 0.30, rhel6 distribution. All running on RHEL 6.5
>
> Enclosed you will find some sample logs from that environment:
>
> 9.12.15_jrnl_size.txt – results from du command
>
> Qpid-stat-q.txt – results from qpid-stat –q
>
> qpid_vsi_qpid-tool-result.txt, qpid_request_qpid-tool-result.txt,
> qpid_public_qpid-tool-result.txt – results from qpid-tool, show #
>
> The question is, why does it happen, and if/how we can prevent it ?
>
> *Best regards*
>
> **
>
> *FARA*
>
> *Adrian Cisło*
>
> Software Engineer
>
> Mobile: +48790618366**
>
> E-mail : adrian.cislo@fara.no <ma...@fara.no>__
>
> /TECHNOLOGY IN MOTION///
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
> For additional commands, e-mail: dev-help@qpid.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org