You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Bill Freeman <ke...@gmail.com> on 2014/03/05 22:55:39 UTC
Queue's various size limits in the C++ broker.
I've a few concerns about my understanding of the real meaning of the
various limits on queue content.
First, let me apologize that my questions need to be interpreted in the
context of the 0.18 release (upgrades here happen slowly, if at all).
Please tell me where I've got it wrong below:
1. Limits on the number of messages in a queue.
I presume that --max-queue-count (qpid.max_count argument in the queue
properties from QMF) works as a limit against the msgDepth statistic from
QMF
msgDepth is always equivalent to (msgTotalEnqueues - msgTotalDequeues).
If a message comes in when msgDepth is already (at least) qpid.max_count,
then the sender receives an exception, the message isn't queued, and the
broker continues to run normally (even if it is part of a cluster).
If --max-queue-count is not specified when a queue is created, then
qpid.max_count doesn't show up in the arguments property, and msgDepth is
unconstrained.
2. Limits on the number of bytes in a queue
I presume that --max-queue-size (qpid.max_size argument in the queue
properties) works as a limit against the byteDepth statistic.
byteDepth is always equivalent to (byteTotalEnqueues - byteTotalDequeues),
even for durable queues where the message body may only exist in the file,
and contrary to some documentation of --mas-queue-size that implies that it
only applies to the in RAM portion of queued messages.
If a message comes in when byteDepth is already (at least) qpid.max_size,
then the sender receives an exception, the message isn't queued, and the
broker continues to run normally (even if it is part of a cluster).
If --max-queue-size is not specified when a queue is created, then
qpid.max_size doesn't show up in the arguments property, and byteDepth is
unconstrained.
3. Limits on the persistence file. (You will find some questions here, and
probably lots of misconceptions.)
--file-count and --file-size (qpid.file_count and qpid.file_size in the QMF
argument property), along with an apparently fixed "page" size of 64kiB,
determine the disk space available to persist this queue. Each of
file_count files is file_size pages.
Are these files pre-allocated, or do they expand to these limits as
messages get persisted?
Messages are not perfectly packed in these files, so the number of bytes of
message that fills this space may be less than the total bytes of disk used.
Is this difference (last paragraph) because of individual message
alignments, or because a message can't be split across files in the set, or
because consumed messages taken from the middle of a file may have that
space re-used by a smaller message, or some combination?
These files only exist for queues marked durable. Specifying --file-count
and --file-size for a non-durable queue is ignored. Is this restriction
implemented in the broker, in qpid-config, or both?
If I add a durable queue (with qpid-config) but don't specify --file-count
and --file-size, I still get qpid.file_count = 8 and qpid.file_size = 12,
as a default size.
I guess that this default sizing is implemented in qpid-config, because a
home grown tool here manages to make queues marked durable, but without
qpid.file_count or qpid.file_size in the arguments property. Do these
queues have persistence space, or are they effectively not durable? If the
broker provides space for this case, how much?
I'm guessing that the file space doesn't directly limit some combination of
statistics, since, for example (bytePersistEnqueues - bytePersistDequeues)
may be somewhat short of the total disk size when the disk space, due to
imperfect packing, becomes full.
When the disk space fills, and the broker is part of a cluster, it shuts
down, rather than refusing the message with an exception to the sender,
over concerns about correct replication to the other cluster members. Or
does this only happen when a replication operation over-fills the file
space?
I see some non-durable queues with non-zero values for
{msg,bytes}Persist{En,De}queues. Is this because messages marked
persistent (durable) is what is actually measured by these parameters, even
when the queue is non-durable, and the messages aren't actually persisted?
Thanks in advance for any clue stick whacking.
Bill
Re: Queue's various size limits in the C++ broker.
Posted by Bill Freeman <ke...@gmail.com>.
Gordon,
Thanks again. This has been a great help.
Bill
Re: Queue's various size limits in the C++ broker.
Posted by Gordon Sim <gs...@redhat.com>.
On 03/06/2014 02:58 PM, Bill Freeman wrote:
> On Thu, Mar 6, 2014 at 5:17 AM, Gordon Sim <gs...@redhat.com> wrote:
>> Can you point me at the misleading docs and I'll see if I can make it
>> clearer?
>
>
> The spot I was thinking of is in the output of qpid-config -h:
>
> --max-queue-size=<n>
> Maximum in-memory queue size as bytes
>
Yes, that is indeed a bit vague and imprecise now you mention it. I
think the wording stems from the (now removed) flow-to-disk policy,
where the queue can grow above the maximum size, but that requires
messages to be 'released' from memory and retrieved from disk when
needed. I.e. in that case it wasn't really a maximum limit, it was a
level at which certain behaviour was triggered.
[...]
> Ah, yes, now that I'm looking for them:
> --default-queue-limit
> --num-jfiles
> --jfile-size-pgs
> And, since we're not setting them, it's good that qpidd --help gives the
> default defaults.
>
> I wonder if I can (for yet another tool I'm writing that tries to display
> percent full in several categories) get the current default (whether
> default default of configured) using QMF? I guess I'll poke around in the
> broker agent class.
I believe there is a QMF class for the store. Using qpid-tool try
`schema store` to see what it has. A quick glance on trunk shows
"defaultInitialFileCount and "defaultDataFileSize" both being available.
You can also get the value on the various journals I think.
[...]
> I guess that I tried to abbreviate that question too much. (Either that or
> I'm too dense to follow your answer.)
No, that would be me not reading the question properly! Sorry!
> msgPersistEnqueues, msgPersistDequeues, bytePersistEnqueues, and
> bytePersistDequeues are queue statistics, not things in the argumenst queue
> property. I had assumed that they indicated the number and total size of
> messages which had been persisted, so I was surprised to see non-zero
> values for a non-durable queue, none of whose content is persisted.
Looking at the code, those are set purely based on whether the *message*
is marked as persistent, regardless of whether the queue is actually
durable (and therefore actually persisting the messages).
[...]
> And now that I think about it, is (bytePersistEnqueues -
> bytePersistDequeues) the aggragate of the original (on the wire) size of
> the messages, as opposed to the size (if the queue is durable) that they
> consume on disk (since this differs).
In 0.18 its not quite the former. On trunk (I think from 0.26 or so) it
*is* more or less the size on the wire. It isn't the size on disk.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Queue's various size limits in the C++ broker.
Posted by Bill Freeman <ke...@gmail.com>.
Gordon,
First, thank you so much for your response.
Some stuff elided:
On Thu, Mar 6, 2014 at 5:17 AM, Gordon Sim <gs...@redhat.com> wrote:
> On 03/05/2014 09:55 PM, Bill Freeman wrote:
>
...
> even for durable queues where the message body may only exist in the file,
>> and contrary to some documentation of --mas-queue-size that implies that
>> it
>> only applies to the in RAM portion of queued messages.
>>
>
> Can you point me at the misleading docs and I'll see if I can make it
> clearer?
The spot I was thinking of is in the output of qpid-config -h:
--max-queue-size=<n>
Maximum in-memory queue size as bytes
...
If --max-queue-size is not specified when a queue is created, then
> qpid.max_size doesn't show up in the arguments property, and byteDepth is
> unconstrained.
>
Actually there is a default max-queue-size, which is set at the broker
> level and applies to all queues that don't set an explicit limit. An
> explicit limit of 0 will give an unconstrained queue.
>
>
and
>
> I guess that this default sizing is implemented in qpid-config, because a
>> home grown tool here manages to make queues marked durable, but without
>> qpid.file_count or qpid.file_size in the arguments property.
>>
>
> That seems to have been changed since 0.18; qpid-config no longer sets
> defaults itself (it will rely on the stores own internal defaults if none
> are specified)
>
> Ah, yes, now that I'm looking for them:
--default-queue-limit
--num-jfiles
--jfile-size-pgs
And, since we're not setting them, it's good that qpidd --help gives the
default defaults.
I wonder if I can (for yet another tool I'm writing that tries to display
percent full in several categories) get the current default (whether
default default of configured) using QMF? I guess I'll poke around in the
broker agent class.
...
I see some non-durable queues with non-zero values for
> {msg,bytes}Persist{En,De}queues. Is this because messages marked
> persistent (durable) is what is actually measured by these parameters, even
> when the queue is non-durable, and the messages aren't actually persisted?
>
The arguments reported by QMF are those that were passed in when the queue
> was created, even if they were not understood or were ignored.
>
>
I guess that I tried to abbreviate that question too much. (Either that or
I'm too dense to follow your answer.)
msgPersistEnqueues, msgPersistDequeues, bytePersistEnqueues, and
bytePersistDequeues are queue statistics, not things in the argumenst queue
property. I had assumed that they indicated the number and total size of
messages which had been persisted, so I was surprised to see non-zero
values for a non-durable queue, none of whose content is persisted. This
led me to rethink that they referred to messages that were marked durable,
whether or not the queue is durable. In a durable queue, the indicated
messages would have been persisted. In a non-durable queue they would not
have been persisted. I was wondering if this last interpretation is
correct.
And now that I think about it, is (bytePersistEnqueues -
bytePersistDequeues) the aggragate of the original (on the wire) size of
the messages, as opposed to the size (if the queue is durable) that they
consume on disk (since this differs).
Thanks again,
Bill
Re: Queue's various size limits in the C++ broker.
Posted by Gordon Sim <gs...@redhat.com>.
On 03/05/2014 09:55 PM, Bill Freeman wrote:
> I've a few concerns about my understanding of the real meaning of the
> various limits on queue content.
>
> First, let me apologize that my questions need to be interpreted in the
> context of the 0.18 release (upgrades here happen slowly, if at all).
>
> Please tell me where I've got it wrong below:
>
> 1. Limits on the number of messages in a queue.
>
> I presume that --max-queue-count (qpid.max_count argument in the queue
> properties from QMF) works as a limit against the msgDepth statistic from
> QMF
>
> msgDepth is always equivalent to (msgTotalEnqueues - msgTotalDequeues).
Correct
> If a message comes in when msgDepth is already (at least) qpid.max_count,
> then the sender receives an exception, the message isn't queued, and the
> broker continues to run normally (even if it is part of a cluster).
Yes, that's the default behaviour. You can set the policy however if you
wish e.g. the oldest messages to be dropped to make room for the new ones.
> If --max-queue-count is not specified when a queue is created, then
> qpid.max_count doesn't show up in the arguments property, and msgDepth is
> unconstrained.
Correct
> 2. Limits on the number of bytes in a queue
>
> I presume that --max-queue-size (qpid.max_size argument in the queue
> properties) works as a limit against the byteDepth statistic.
>
> byteDepth is always equivalent to (byteTotalEnqueues - byteTotalDequeues),
Correct
> even for durable queues where the message body may only exist in the file,
> and contrary to some documentation of --mas-queue-size that implies that it
> only applies to the in RAM portion of queued messages.
Can you point me at the misleading docs and I'll see if I can make it
clearer?
> If a message comes in when byteDepth is already (at least) qpid.max_size,
> then the sender receives an exception, the message isn't queued, and the
> broker continues to run normally (even if it is part of a cluster).
Yes, although as above, a different policy can be chosen for how to
handle the case where the limit is reached.
> If --max-queue-size is not specified when a queue is created, then
> qpid.max_size doesn't show up in the arguments property, and byteDepth is
> unconstrained.
Actually there is a default max-queue-size, which is set at the broker
level and applies to all queues that don't set an explicit limit. An
explicit limit of 0 will give an unconstrained queue.
> 3. Limits on the persistence file. (You will find some questions here, and
> probably lots of misconceptions.)
>
> --file-count and --file-size (qpid.file_count and qpid.file_size in the QMF
> argument property), along with an apparently fixed "page" size of 64kiB,
> determine the disk space available to persist this queue. Each of
> file_count files is file_size pages.
>
> Are these files pre-allocated, or do they expand to these limits as
> messages get persisted?
Currently they are preallocated. There is a new store (the so called
'linear store') that allocates files to any queue from a central list,
and can expand as needed. This is I think available for preview in 0.26,
but is still undergoing testing and fixing.
> Messages are not perfectly packed in these files, so the number of bytes of
> message that fills this space may be less than the total bytes of disk used.
>
> Is this difference (last paragraph) because of individual message
> alignments, or because a message can't be split across files in the set, or
> because consumed messages taken from the middle of a file may have that
> space re-used by a smaller message, or some combination?
Its due to the alignment of individual messages, the extra records that
are added to signal dequeue etc. The journal in the existing store
('legacy store' so called) is treated as a ring buffer, so removing
messages from the middle does not actually add any usable space.
> These files only exist for queues marked durable. Specifying --file-count
> and --file-size for a non-durable queue is ignored. Is this restriction
> implemented in the broker, in qpid-config, or both?
In the broker only I believe. Its not really a restriction so much as
the fact that the options won't be looked at if the queue is not durable.
> If I add a durable queue (with qpid-config) but don't specify --file-count
> and --file-size, I still get qpid.file_count = 8 and qpid.file_size = 12,
> as a default size.
>
> I guess that this default sizing is implemented in qpid-config, because a
> home grown tool here manages to make queues marked durable, but without
> qpid.file_count or qpid.file_size in the arguments property.
That seems to have been changed since 0.18; qpid-config no longer sets
defaults itself (it will rely on the stores own internal defaults if
none are specified)
> Do these
> queues have persistence space, or are they effectively not durable? If the
> broker provides space for this case, how much?
They are durable, the defaults will be as configured for the store (You
can see the options with qpidd --help, provided store module is loaded).
> I'm guessing that the file space doesn't directly limit some combination of
> statistics, since, for example (bytePersistEnqueues - bytePersistDequeues)
> may be somewhat short of the total disk size when the disk space, due to
> imperfect packing, becomes full.
Actually in my experience the journals capacity can be significantly
short of the max queue size. This is in large part due to the nature of
the journal and all the overheads, alignments and paddings etc, but is
also because the size of the message on disk is larger than the size
used to increment the aggregate depth in bytes (this has been adjusted
in 0.26 to include headers etc, so gets a little closer).
> When the disk space fills, and the broker is part of a cluster, it shuts
> down, rather than refusing the message with an exception to the sender,
> over concerns about correct replication to the other cluster members. Or
> does this only happen when a replication operation over-fills the file
> space?
The issue is that reaching the journal capacity isn't entirely
deterministic. It can happen on one node but not the others. When the
broker gets an error from any node that is not also raised on all the
others, it shuts down all those that had the error.
> I see some non-durable queues with non-zero values for
> {msg,bytes}Persist{En,De}queues. Is this because messages marked
> persistent (durable) is what is actually measured by these parameters, even
> when the queue is non-durable, and the messages aren't actually persisted?
The arguments reported by QMF are those that were passed in when the
queue was created, even if they were not understood or were ignored.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org