You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Craig Forbes <cp...@nemeannetworks.com> on 2010/09/01 16:55:56 UTC
Problem with persitent store and routing (c++ broker)
A quick summary of the problem.
I have a remote c++ broker with a persistent queue. There is a push route
on the remote broker to send messages in the persistent queue to a central
broker.
When the persistent store's journal is full (actually 80% full) and won't
accept more messages. The push route stops relaying the messages in the
queue. If I stop and restart the broker the queue route starts working
again and the queue is drained.
Has anyone run into this before? Is it a known issue?
In case you want more details here they are:
A remote C++ broker with a persistent queue and limit-policy=ring
A central C++ broker with a persistent queue and limit-policy=ring
I have a route setup on the remote system to route all messages from the
remote queue to the central queue
I configured the queue and route on the remote system like this:
qpid-config add queue QUEUE_NAME --durable --limit-policy ring
qpid-route --transport ssl --durable --src-local queue add amqps://central
localhost EXCHANGE_NAME QUEUE_NAME
The idea is the remote machine has several clients submitting to the its
broker which then sends them on the the central system, and queues them
locally in the even the central broker can not be reached.
This all works fine.
The problem occurs when the journal on the remote machine fills up. When
that occurs I see error messages about the queue being full and the
inter-broker link disconnecting:
13:06:40 warning Client closed connection with 501: Enqueue capacity
threshold exceeded on queue "QUEUE_NAME". (JournalImpl.cpp:621)
13:06:40 warning Inter-broker link disconnected from central:5671
13:06:45 warning Client closed connection with 501: Enqueue capacity
threshold exceeded on queue "QUEUE_NAME". (JournalImpl.cpp:621)
13:06:45 warning Inter-broker link disconnected from central:5671
Thanks for your help,
-Craig
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Problem with persitent store and routing (c++ broker)
Posted by Kim van der Riet <ki...@redhat.com>.
On Thu, 2010-09-02 at 09:35 +0100, Gordon Sim wrote:
> On 09/01/2010 03:55 PM, Craig Forbes wrote:
> > A quick summary of the problem.
> >
> > I have a remote c++ broker with a persistent queue. There is a push
> > route on the remote broker to send messages in the persistent queue
> to a
> > central broker.
> >
> > When the persistent store's journal is full (actually 80% full) and
> > won't accept more messages. The push route stops relaying the
> messages
> > in the queue. If I stop and restart the broker the queue route
> starts
> > working again and the queue is drained.
> >
> > Has anyone run into this before? Is it a known issue?
>
> Its expected that if the queue to which messages are being sent fills
> up
> then the route over which they are being sent will be interrupted by
> the
> exception. I believe it should periodically retry however and that
> seems
> to be borne our by the log below.
>
> What seems odd is why restarting the broker fixes things... (Kim,
> might
> that be due to better compaction of the journal on restart?)
>
Sorry, I missed this question, and I have not answered it.
There is no compaction on the store on restart. For a regular queue,
when an enqueue threshold is reached, only the connection responsible
for publishing the message which exceeded the threshold is shut down.
Other connections which consume messages continue without interruption,
thereby creating space as long as they run.
I am uncertain how the behavior for LVQ changes this, however, as
messages are not consumed in the "normal" way in this case. From a store
perspective, space is made as long as messages are being dequeued. Is it
possible that older values in the store are being discarded on recovery?
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Problem with persitent store and routing (c++ broker)
Posted by Craig Forbes <cp...@nemeannetworks.com>.
--On Thursday, September 02, 2010 09:35:53 AM +0100 Gordon Sim
<gs...@redhat.com> wrote:
> On 09/01/2010 03:55 PM, Craig Forbes wrote:
>> I configured the queue and route on the remote system like this:
>> qpid-config add queue QUEUE_NAME --durable --limit-policy ring
>
> The above doesn't set an explicit limit, so the default limit will be
> used (as controlled via qpidd's --default-queue-limit option). I suspect
> that is higher than the limit for the default journal size.
>
> If you use a max-size or max-count here to set an explicit limit that
> keeps the queue within the journals limits then the error should be
> avoided (as the older messages will be overwritten by newer ones, which I
> presume is what you want).
Ah, this is key. My testing included setting a queue size limit but it
seems that it didn't get moved into the final configuration.
Thanks,
-Craig
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Problem with persitent store and routing (c++ broker)
Posted by Gordon Sim <gs...@redhat.com>.
On 09/01/2010 03:55 PM, Craig Forbes wrote:
> A quick summary of the problem.
>
> I have a remote c++ broker with a persistent queue. There is a push
> route on the remote broker to send messages in the persistent queue to a
> central broker.
>
> When the persistent store's journal is full (actually 80% full) and
> won't accept more messages. The push route stops relaying the messages
> in the queue. If I stop and restart the broker the queue route starts
> working again and the queue is drained.
>
> Has anyone run into this before? Is it a known issue?
Its expected that if the queue to which messages are being sent fills up
then the route over which they are being sent will be interrupted by the
exception. I believe it should periodically retry however and that seems
to be borne our by the log below.
What seems odd is why restarting the broker fixes things... (Kim, might
that be due to better compaction of the journal on restart?)
> In case you want more details here they are:
>
> A remote C++ broker with a persistent queue and limit-policy=ring
>
> A central C++ broker with a persistent queue and limit-policy=ring
>
> I have a route setup on the remote system to route all messages from the
> remote queue to the central queue
>
> I configured the queue and route on the remote system like this:
> qpid-config add queue QUEUE_NAME --durable --limit-policy ring
The above doesn't set an explicit limit, so the default limit will be
used (as controlled via qpidd's --default-queue-limit option). I suspect
that is higher than the limit for the default journal size.
If you use a max-size or max-count here to set an explicit limit that
keeps the queue within the journals limits then the error should be
avoided (as the older messages will be overwritten by newer ones, which
I presume is what you want).
> qpid-route --transport ssl --durable --src-local queue add
> amqps://central localhost EXCHANGE_NAME QUEUE_NAME
>
> The idea is the remote machine has several clients submitting to the its
> broker which then sends them on the the central system, and queues them
> locally in the even the central broker can not be reached.
>
> This all works fine.
>
> The problem occurs when the journal on the remote machine fills up. When
> that occurs I see error messages about the queue being full and the
> inter-broker link disconnecting:
>
> 13:06:40 warning Client closed connection with 501: Enqueue capacity
> threshold exceeded on queue "QUEUE_NAME". (JournalImpl.cpp:621)
>
> 13:06:40 warning Inter-broker link disconnected from central:5671
>
> 13:06:45 warning Client closed connection with 501: Enqueue capacity
> threshold exceeded on queue "QUEUE_NAME". (JournalImpl.cpp:621)
>
> 13:06:45 warning Inter-broker link disconnected from central:5671
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org