resizing persistent queue journal

[Charles Woerner <cw...@demandbase.com>]

Hello,

I have 3 questions relating to persistent queue journal sizing:

1) What is the recommended way to expand the size of a queue's  
journal?  Ie. if I reach the 80% threshhold is there a way to expand  
the journal size by redefining the num-jfiles and jfile-size-pgs of an  
existing durable queue?

2) The qpid source makes reference to an auto_expand property, but I  
don't see any documentation on that.  Can I set that property on an  
existing queue in order to coax it to expand itself?

3) Lastly, the /etc/qpidd.conf and qpidd startup parameters pertaining  
to journal size (num-jfiles, jfile-size-pgs) don't seem to have any  
affect for me.  Using them as addQueueOptions to queue-config, however  
does work.  Is the expected behavior that new queues should inherit  
those properties if specific num-jfiles and jfile-size-pgs are not  
explicitly declared at queue creation time?
__

Charles Woerner  | cwoerner@demandbase.com |   demandbase   |   
415.683.2669

Re: resizing persistent queue journal

[Charles Woerner <CW...@demandbase.com>]

Kim,

Thanks for the helpful information.  I'll take a look at "resize" and  
double check the behavior I saw on queue_declare()'s with respect to  
the qpidd.conf store size options and post the result if I confirm  
that the behavior I'm seeing is different from your expectation.
__

Charles Woerner  | cwoerner@demandbase.com |   demandbase

Re: resizing persistent queue journal

[Kim van der Riet <ki...@redhat.com>]

On Fri, 2010-01-08 at 17:39 -0800, Charles Woerner wrote:
> Hello,
> 
> I have 3 questions relating to persistent queue journal sizing:
> 
> 1) What is the recommended way to expand the size of a queue's  
> journal?  Ie. if I reach the 80% threshhold is there a way to expand  
> the journal size by redefining the num-jfiles and jfile-size-pgs of an  
> existing durable queue?

Not for an existing journal which is running. However, there is a
command-line utility that can take an existing journal of a stopped
broker and resize it. This is a new feature and is being currently
tested. If you are using the svn trunk, you can try it: resize. It will
move the current journal into a backup folder, then create a new journal
of a different size (either bigger or smaller) and transfer all still
undequeued records from the old journal into the new one. Once this is
done, the broker can be restarted.

> 
> 2) The qpid source makes reference to an auto_expand property, but I  
> don't see any documentation on that.  Can I set that property on an  
> existing queue in order to coax it to expand itself?

This is not implemented. There is a good deal of auto-expand framework
in the code, but the core bits which allow the expansion are not
included at this time. That is the reason there is no documentation.

> 
> 3) Lastly, the /etc/qpidd.conf and qpidd startup parameters pertaining  
> to journal size (num-jfiles, jfile-size-pgs) don't seem to have any  
> affect for me.  Using them as addQueueOptions to queue-config, however  
> does work.  Is the expected behavior that new queues should inherit  
> those properties if specific num-jfiles and jfile-size-pgs are not  
> explicitly declared at queue creation time?

Using command-line parameters and/or the qpidd.conf file sets the
default for all new persistent queues. However, the parameters of
existing recovered queues must take precedence over the new parameters.
If you start a broker with existing persistent queues on disk, then the
parameters of the existing recovered queue will override the default
(with a warning in the broker log - make sure your broker is set verbose
enough to print warnings - try "--log-enable info+", which will give a
good deal of information about the creation and recovery of persistent
queues). Thus if you are using the same persistent queue names and these
are recovered (even if there are no messages on them), then the old size
parameters will stick. To overcome this problem, either:

a) get your application to delete the queue then recreate it;

b) stop the broker, then delete the store directory before restarting or
use a different data-dir or store-dir parameter;

c) use new queue names.

If a newly created persistent queue ignores the command-line parameters
or the config file, then this could be a bug.

> __
> 
> Charles Woerner  | cwoerner@demandbase.com |   demandbase   |   
> 415.683.2669
> 
> 
> 
> 
> 
> 



---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org