You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Bruno Matos <br...@paradigmaxis.pt> on 2011/02/08 11:02:01 UTC
Re: Journal with LVQ
On 2011/01/19, at 14:46, Gordon Sim wrote:
> On 01/18/2011 03:31 PM, Bruno Matos wrote:
>>
>> On 2011/01/12, at 17:33, Gordon Sim wrote:
>>
>>> On 01/11/2011 04:30 PM, Bruno Matos wrote:
>>>> I'm publishing messages to a durable LVQ, when I run the
>>>> publisher for
>>>> the first time the journal stays almost full, then I send the
>>>> exact same
>>>> messages, expecting the old ones to be replaced, but after some
>>>> time the
>>>> connection is closed with "Enqueue capacity threshold exceeded on
>>>> queue". With qpid-tool I can see that msgDepth and byteDepth are
>>>> exactly
>>>> the same before and after the second run. Is this the expected
>>>> behavior?
>>>
>>> Depends on what you mean by 'expected' :-)
>>>
>>> It is certainly the case that the linux journal will reach capacity
>>> with different queue depths, depending on exact sequences of
>>> changes.
>>> i.e. the capacity is not simply a function of the final state, but
>>> also of the path to get there. In the second case there would be
>>> additional dequeue records and the file boundaries might not align
>>> the
>>> same way etc, so the journal space can be used up for fewer enqueued
>>> messages.
>>
>>
>> Thank you Gordon.
>> I think I can say that's the expected to me... :) Although I need to
>> know what journal size is effectively in use, can be a percentage, is
>> that possible? I have a stored queue for browsing, like a dictionary,
>> that is updated every night, and I want to reset it only if it
>> reaches
>> to a given percentage of journal size.
>
> Not sure if I understand the question correctly, but there is no
> simple way of determining the minimum journal size needed to
> guarantee a specific queue depth. The rule I use is to size the
> journal much larger than I think I need (e.g. 2x the queue depth).
> Kim has some text somewhere on the calculations involved I believe.
>
> Its also worth then setting a queue policy on the queue with the
> depth fixed (and significantly less than the journal capacity). That
> way you get a cleaner error when you hit the limit.
>
Sorry the delay.
I was asking if there is a way to know the used size of the journal in
a given time. When I run my publisher for the first time, and in qpid-
stat the queue size 2m in bytes, how can I know the effectively used
journal?
Tia.
--
Bruno Matos
bruno.matos@paradigmaxis.pt
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Journal with LVQ
Posted by Alan Conway <ac...@redhat.com>.
On 02/15/2011 10:31 AM, Kim van der Riet wrote:
> On Tue, 2011-02-15 at 09:08 -0500, Alan Conway wrote:
>>>> I was asking if there is a way to know the used size of the journal
>> in a
>>>> given time. When I run my publisher for the first time, and in
>> qpid-stat
>>>> the queue size 2m in bytes, how can I know the effectively used
>> journal?
>>>
>>> The calculation is quite complex and there tend to be lots of
>> variables (exact
>>> timings of dequeues v. enqueues, relative messages sizes etc). in
>> practical
>>> terms I have not found a way to accurately estimate the journal size
>> required
>>> and so always greatly inflate it. It is an area I think we need to
>> make simpler
>>> in some way.
>>>
>>
>> Perhaps it would be useful to add some management instrumentation to
>> the
>> journals, so it is at least easy to see how much space is actually
>> used when you
>> run a test (or indeed in production so you can see if you're getting
>> close to
>> the limit)
>>
> This has been discussed. It is likely that a QMF function call would be
> added which would calculate the amount of space left in the store when
> it is called. However, to do this continually (it through
> instrumentation) would likely be expensive and could slow the store down
> significantly.
>
Doesn't that depend on how often you do it? E.g. could the user to configure an
alert every 10 minutes, or if the free space goes below 10% etc.?
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Journal with LVQ
Posted by Kim van der Riet <ki...@redhat.com>.
On Tue, 2011-02-15 at 09:08 -0500, Alan Conway wrote:
> >> I was asking if there is a way to know the used size of the journal
> in a
> >> given time. When I run my publisher for the first time, and in
> qpid-stat
> >> the queue size 2m in bytes, how can I know the effectively used
> journal?
> >
> > The calculation is quite complex and there tend to be lots of
> variables (exact
> > timings of dequeues v. enqueues, relative messages sizes etc). in
> practical
> > terms I have not found a way to accurately estimate the journal size
> required
> > and so always greatly inflate it. It is an area I think we need to
> make simpler
> > in some way.
> >
>
> Perhaps it would be useful to add some management instrumentation to
> the
> journals, so it is at least easy to see how much space is actually
> used when you
> run a test (or indeed in production so you can see if you're getting
> close to
> the limit)
>
This has been discussed. It is likely that a QMF function call would be
added which would calculate the amount of space left in the store when
it is called. However, to do this continually (it through
instrumentation) would likely be expensive and could slow the store down
significantly.
That said, we want to make this accessible and are looking for smart
ways to do it.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Journal with LVQ
Posted by Bruno Matos <br...@paradigmaxis.pt>.
On 2011/02/15, at 14:39, Gordon Sim wrote:
> On 02/15/2011 02:08 PM, Alan Conway wrote:
>> Perhaps it would be useful to add some management instrumentation
>> to the
>> journals, so it is at least easy to see how much space is actually
>> used
>> when you run a test (or indeed in production so you can see if you're
>> getting close to the limit)
>
> Good idea
>
It would be great.
--
Bruno Matos
bruno.matos@paradigmaxis.pt
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Journal with LVQ
Posted by Gordon Sim <gs...@redhat.com>.
On 02/15/2011 02:08 PM, Alan Conway wrote:
> Perhaps it would be useful to add some management instrumentation to the
> journals, so it is at least easy to see how much space is actually used
> when you run a test (or indeed in production so you can see if you're
> getting close to the limit)
Good idea
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Journal with LVQ
Posted by Alan Conway <ac...@redhat.com>.
On 02/15/2011 07:01 AM, Gordon Sim wrote:
> On 02/08/2011 10:02 AM, Bruno Matos wrote:
>>
>> On 2011/01/19, at 14:46, Gordon Sim wrote:
>>
>>> On 01/18/2011 03:31 PM, Bruno Matos wrote:
>>>>
>>>> On 2011/01/12, at 17:33, Gordon Sim wrote:
>>>>
>>>>> On 01/11/2011 04:30 PM, Bruno Matos wrote:
>>>>>> I'm publishing messages to a durable LVQ, when I run the publisher for
>>>>>> the first time the journal stays almost full, then I send the exact
>>>>>> same
>>>>>> messages, expecting the old ones to be replaced, but after some
>>>>>> time the
>>>>>> connection is closed with "Enqueue capacity threshold exceeded on
>>>>>> queue". With qpid-tool I can see that msgDepth and byteDepth are
>>>>>> exactly
>>>>>> the same before and after the second run. Is this the expected
>>>>>> behavior?
>>>>>
>>>>> Depends on what you mean by 'expected' :-)
>>>>>
>>>>> It is certainly the case that the linux journal will reach capacity
>>>>> with different queue depths, depending on exact sequences of changes.
>>>>> i.e. the capacity is not simply a function of the final state, but
>>>>> also of the path to get there. In the second case there would be
>>>>> additional dequeue records and the file boundaries might not align the
>>>>> same way etc, so the journal space can be used up for fewer enqueued
>>>>> messages.
>>>>
>>>>
>>>> Thank you Gordon.
>>>> I think I can say that's the expected to me... :) Although I need to
>>>> know what journal size is effectively in use, can be a percentage, is
>>>> that possible? I have a stored queue for browsing, like a dictionary,
>>>> that is updated every night, and I want to reset it only if it reaches
>>>> to a given percentage of journal size.
>>>
>>> Not sure if I understand the question correctly, but there is no
>>> simple way of determining the minimum journal size needed to guarantee
>>> a specific queue depth. The rule I use is to size the journal much
>>> larger than I think I need (e.g. 2x the queue depth). Kim has some
>>> text somewhere on the calculations involved I believe.
>>>
>>> Its also worth then setting a queue policy on the queue with the depth
>>> fixed (and significantly less than the journal capacity). That way you
>>> get a cleaner error when you hit the limit.
>>>
>>
>> Sorry the delay.
>>
>> I was asking if there is a way to know the used size of the journal in a
>> given time. When I run my publisher for the first time, and in qpid-stat
>> the queue size 2m in bytes, how can I know the effectively used journal?
>
> The calculation is quite complex and there tend to be lots of variables (exact
> timings of dequeues v. enqueues, relative messages sizes etc). in practical
> terms I have not found a way to accurately estimate the journal size required
> and so always greatly inflate it. It is an area I think we need to make simpler
> in some way.
>
Perhaps it would be useful to add some management instrumentation to the
journals, so it is at least easy to see how much space is actually used when you
run a test (or indeed in production so you can see if you're getting close to
the limit)
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Journal with LVQ
Posted by Gordon Sim <gs...@redhat.com>.
On 02/08/2011 10:02 AM, Bruno Matos wrote:
>
> On 2011/01/19, at 14:46, Gordon Sim wrote:
>
>> On 01/18/2011 03:31 PM, Bruno Matos wrote:
>>>
>>> On 2011/01/12, at 17:33, Gordon Sim wrote:
>>>
>>>> On 01/11/2011 04:30 PM, Bruno Matos wrote:
>>>>> I'm publishing messages to a durable LVQ, when I run the publisher for
>>>>> the first time the journal stays almost full, then I send the exact
>>>>> same
>>>>> messages, expecting the old ones to be replaced, but after some
>>>>> time the
>>>>> connection is closed with "Enqueue capacity threshold exceeded on
>>>>> queue". With qpid-tool I can see that msgDepth and byteDepth are
>>>>> exactly
>>>>> the same before and after the second run. Is this the expected
>>>>> behavior?
>>>>
>>>> Depends on what you mean by 'expected' :-)
>>>>
>>>> It is certainly the case that the linux journal will reach capacity
>>>> with different queue depths, depending on exact sequences of changes.
>>>> i.e. the capacity is not simply a function of the final state, but
>>>> also of the path to get there. In the second case there would be
>>>> additional dequeue records and the file boundaries might not align the
>>>> same way etc, so the journal space can be used up for fewer enqueued
>>>> messages.
>>>
>>>
>>> Thank you Gordon.
>>> I think I can say that's the expected to me... :) Although I need to
>>> know what journal size is effectively in use, can be a percentage, is
>>> that possible? I have a stored queue for browsing, like a dictionary,
>>> that is updated every night, and I want to reset it only if it reaches
>>> to a given percentage of journal size.
>>
>> Not sure if I understand the question correctly, but there is no
>> simple way of determining the minimum journal size needed to guarantee
>> a specific queue depth. The rule I use is to size the journal much
>> larger than I think I need (e.g. 2x the queue depth). Kim has some
>> text somewhere on the calculations involved I believe.
>>
>> Its also worth then setting a queue policy on the queue with the depth
>> fixed (and significantly less than the journal capacity). That way you
>> get a cleaner error when you hit the limit.
>>
>
> Sorry the delay.
>
> I was asking if there is a way to know the used size of the journal in a
> given time. When I run my publisher for the first time, and in qpid-stat
> the queue size 2m in bytes, how can I know the effectively used journal?
The calculation is quite complex and there tend to be lots of variables
(exact timings of dequeues v. enqueues, relative messages sizes etc). in
practical terms I have not found a way to accurately estimate the
journal size required and so always greatly inflate it. It is an area I
think we need to make simpler in some way.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org