You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Fraser Adams <fr...@blueyonder.co.uk> on 2012/08/03 12:42:44 UTC
more memory "leak" woes.
Hello all,
I've previously posted with a problem whereby in a federated set up
we've been seeing brokers eat memory. In that case what was weird was
that when the producer client was pointed to a broker on the same box
we'd start to see problems, but when it was pointed to a broker on a
remote box it largely seemed to be stable. It drove us nuts, but we were
able to park it for a while. We've recently started seeing it again
where we really had to use a co-located broker.
One thing that seems interesting is that it happens most when the
consumer was in some way behaving slower than it should.
We're publishing messages to amq.match and the queue the consumers
receive from is a RING queue, so as far as I can see (intuitively at
least!!) it shouldn't really matter if the consumer is slow - the
circular queue should just overwrite the oldest messages.
One of my colleagues wrote a couple of little programs using the
qpid::client API to reproduce a bursty producer and a slow consumer and
this seems to reproduce the problem - basically despite a circular queue
both qpidd and client-consumer memory consumption grows and I start
swapping madly on my 4G box despite the queue only being 100MB.
I'm more familiar with qpid::messaging, so I tried to reproduce the
problem using that, what's interesting is that my first attempt couldn't
reproduce it - but of course the APIs behave differently. Then I started
wondering about flow control/capacity.
I usually do setCapacity(500) - for no other reason than I think that's
what the default is for the JMS API. Now with that I was using more
qpidd memory than I'd expect with a 100M circular queue, but I then
reduced it to 100 then 10 and realised that the capacity (which I
thought related to prefetch on the client) was affecting both client and
qpidd memory consumption. I also noticed that doing "link: {reliability:
unreliable}" helped.
I tried enabling flow control in the client-consumer to no avail (I
struggle to understand that API!!) - I thought adding
SubscriptionSettings settings;
settings.autoAck = 100;
settings.flowControl = FlowControl::messageCredit(200);
subscription = subscriptions.subscribe(*this, queue, settings);
to my prepareQueue() method would be the way to do it, but that just
seemed to cause my consumer to hang after it had received 200 messages.
So my messaging-consumer can be made to behave in a way that at least
makes some sense, but what concerns me is back to the problem of
federated links - as I say we're seeing terrible resource leaks and I
assume that the federation bridge code is closer to what qpid::client is
doing, so I've no idea if we can configure a federated link to "honour"
the behaviour that we'd expect from a circular queue (we use the default
behaviour which *should* we using unreliable messaging).
hope this makes sense. I've attached the producer and two consumers that
I've been using to try this stuff out.
I'd appreciate any thoughts and especially any mitigations, this is
starting to cause us real problems.
MTIA
Fraser.
-------- Original Message --------
Subject: Re: C++ broker memory leak in federated set-up???
Date: Thu, 01 Mar 2012 15:04:04 +0000
From: Gordon Sim <gs...@redhat.com>
Reply-To: users@qpid.apache.org
Organisation: Red Hat UK Ltd, Registered in England and Wales under
Company Registration No. 3798903, Directors: Michael Cunningham (USA),
Mark Hegarty (Ireland), Matt Parsons (USA), Charlie Peters (USA)
To: users@qpid.apache.org
On 02/29/2012 07:07 PM, Fraser Adams wrote:
> Hi All,
> I think that we may have stumbled across a potential memory/resource leak.
>
> We have one particular set up where we have a C++ producer client (using
> qpid::client - don't ask, it's a long story.....) this writes to a 0.8
> broker hosted on the same server. That broker is then federated via a
> queue route to amq.match on another (0.8) broker. The queue route is a
> source route set up via qpid-route -s
>
> We've been having all sorts of fun and games with respect to
> performance, which we've narrowed down to some dodgy networking.
>
> However one of the other effects that we've noticed is that the broker
> co-located with the producer client eats memory. The queue for the queue
> route is 1GB but qpidd eventually grows to ~35GB and sends the whole set
> up into swap.
>
> So with respect to the network problem we're suspecting a dodgy switch
> somewhere, what is interesting is that when we checked with ethtool the
> NIC was reporting half duplex had been negotiated - ouch!!! hence why we
> suspect a dodgy switch somewhere.
>
> Now when the NIC was explicitly set to 100 base/T full duplex our
> performance rocketed and the broker on the producer system appears
> (touch wood) to have stable memory performance.
>
> What I'm suspecting is that the dodgy network link has been causing
> connection drop-outs and the broker is automatically reconnecting (logs
> are confirming this) and I'm thinking that there is a resource leak
> somewhere during the reconnection process.
https://issues.apache.org/jira/browse/QPID-3447 perhaps? Though I
wouldn't have expected that to cause such a large growth in memory.
Your sure there is no backed up queue anywhere?
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: more memory "leak" woes.
Posted by Gordon Sim <gs...@redhat.com>.
On 08/03/2012 06:51 PM, Fraser Adams wrote:
>
>>
>> subscription = subscriptions.subscribe(*this, queue,
>> SubscriptionSettings(FlowControl::messageWindow(100),
>> ACCEPT_MODE_NONE));
> Unfortunately that setting just seemed to cause my consumer to hang
> after getting the first 100 messages.
>
> subscription = subscriptions.subscribe(*this, queue,
> SubscriptionSettings(FlowControl::messageWindow(100)));
>
> seemed to work OK without growing memory. I've no idea why adding
> ACCEPT_MODE_NONE should cause it to misbehave, it should have simply
> removed the need to acknowledge should it not...
Sorry... one more complication with that combination... you will need to
periodically call Session::sendCompletion(), certainly not less then
every 100 messages. This is because if accepts are not required the
auto-ack functionality is disabled and that is where the
sendCompletion() is done autonatically. (Again, this is what you are
avoiding by using the messaging API everyone! :-)
However, from the perspective of delivery records, if you set
ACCEPT_MODE_NONE, you can leave the flow control as unlimited and you
should still not build up any server side memory.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: more memory "leak" woes.
Posted by Fraser Adams <fr...@blueyonder.co.uk>.
>
> subscription = subscriptions.subscribe(*this, queue,
> SubscriptionSettings(FlowControl::messageWindow(100),
> ACCEPT_MODE_NONE));
Unfortunately that setting just seemed to cause my consumer to hang
after getting the first 100 messages.
subscription = subscriptions.subscribe(*this, queue,
SubscriptionSettings(FlowControl::messageWindow(100)));
seemed to work OK without growing memory. I've no idea why adding
ACCEPT_MODE_NONE should cause it to misbehave, it should have simply
removed the need to acknowledge should it not...
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: more memory "leak" woes.
Posted by Gordon Sim <gs...@redhat.com>.
On 08/04/2012 11:03 AM, Fraser Adams wrote:
> The federation links that we are using are source/push routes e.g. set
> up using qpid-route -s
>
> I wonder if that would have any bearing, so the scenario is really a
> bursty producer much like the client-producer example, writing to
> amq.match on a broker co-located on the same host, which has a queue
> route federated to another broker on a different host, which has a
> potentially slow consumer with possibly/probably too much prefetch
> allocated.
>
> So using source routes is it likely then that an end consumer requesting
> too much capacity could cause a problem for the first broker???
Any subscriber requesting a large amount of prefetch could potentially
cause memory growth in the broker they have subscribed to, particularly
if they are slow to process such prefetched messages.
The slowness of a consumer on a queue on the downstream (i.e.
destination) broker to which messages from the upstream (i.e. source)
broker are routed doesn't directly affect the subscription on the
upstream broker.
If it is a slow consumer problem, then it would imply that it is the
handling of messages from the source broker (i.e. routing them through
the amq.match exchange), that is the problem. Comparing the stats from
the source queue and the target exchange might show something if that
were the case (i.e. many more messages delivered from the queue than
routed through the exchange; that of course may require compensating for
the point you start at)
> It (sort
> of) seems odd because if the federation uses unreliable messaging then
> surely that should protect the source broker from the excess capacity -
> but it's very definitely the source broker associated with the producer
> and not the destination broker where be see the issue.
>
>
> Hmmm I wonder Gordon. You've previously said that "federation
> subscriptions do use infinite credit" , so with a source route the
> bridge would behave essentially like a consumer of the queue actually
> deployed on the source broker,
Depends on what you mean; the route appears to the broker as a
subscription from the queue. The push route mainly affects the manner in
which the subscription is issued (it is 'forged' by the broker who sends
the request to itself as if it were the remote peer sending it).
> so if there were any problems at all in
> its communications to the destination broker then it would behave
> exactly like my slow consumer scenario would it not and its memory would
> grow due to the infinite credit? Now the fact that it's unreliable
> messaging might help, but is it correctly implemented for source routes?
The selection of the accept mode is the same code in both types of
routes, so on the face of it at least I can't see why there would be any
difference there.
> You've said before that source routes are the path less travelled, so
> that feels plausible (especially given my experiences with qpid::client
> :-))?
>
> What do you think - do you have any evidence your side or am I just
> clutching at straws??
I can't see any evidence that source routes would cause the behaviour
you are seeing.
If you can reproduce easily, I would suggest (a) once you start seeing
the excessive growth, kill and then re-establish the route and see
whether that resets the growth, (b) compile a later broker in a working
directory somewhere (no need to install) and see if that shows the same
thing.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: more memory "leak" woes.
Posted by Fraser Adams <fr...@blueyonder.co.uk>.
>> If I can get evidence to suggest that the federation "unreliable links"
>> behaviour isn't as effective at mitigating this in 0.8 as it is in later
>> brokers that would be useful, as I say I keep arguing that we should be
>> upgrading to something newer.
>
> I've had a scan through the svn logs and I don't see anything
> obviously broken in 0.8; the lines that allow the message to be
> released are in there. However a lot has changed since and many other
> things could interact.
One more possibly relevant detail in this sorry tale of woe.....
The federation links that we are using are source/push routes e.g. set
up using qpid-route -s
I wonder if that would have any bearing, so the scenario is really a
bursty producer much like the client-producer example, writing to
amq.match on a broker co-located on the same host, which has a queue
route federated to another broker on a different host, which has a
potentially slow consumer with possibly/probably too much prefetch
allocated.
So using source routes is it likely then that an end consumer requesting
too much capacity could cause a problem for the first broker??? It (sort
of) seems odd because if the federation uses unreliable messaging then
surely that should protect the source broker from the excess capacity -
but it's very definitely the source broker associated with the producer
and not the destination broker where be see the issue.
Hmmm I wonder Gordon. You've previously said that "federation
subscriptions do use infinite credit" , so with a source route the
bridge would behave essentially like a consumer of the queue actually
deployed on the source broker, so if there were any problems at all in
its communications to the destination broker then it would behave
exactly like my slow consumer scenario would it not and its memory would
grow due to the infinite credit? Now the fact that it's unreliable
messaging might help, but is it correctly implemented for source routes?
You've said before that source routes are the path less travelled, so
that feels plausible (especially given my experiences with qpid::client
:-))?
What do you think - do you have any evidence your side or am I just
clutching at straws??
Frase
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: more memory "leak" woes.
Posted by Gordon Sim <gs...@redhat.com>.
On 08/03/2012 03:05 PM, Fraser Adams wrote:
> BTW do you know of a good way to allow multiple broker versions to
> co-exist on the same box (short of virtualisation). I always seem to
> need to completely uninstall one version before I can install another
> due to library dependencies. I guess that I could set up a chroot jail,
> but I was wondering if there's a simpler approach?
If you are building yourself, that shouldn't be an issue. You can either
use a prefix when running configure or simply run from the build
directory directly.
If you are using pre-built packages it will likely be a lot trickier.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: more memory "leak" woes.
Posted by Fraser Adams <fr...@blueyonder.co.uk>.
On 03/08/12 14:38, Gordon Sim wrote:
> On 08/03/2012 01:35 PM, Fraser Adams wrote:
>> So if the "unreliable message" behaviour isn't behaving as expected in
>> earlier brokers (say 0.8) and we have a slow consumer hanging off the
>> other side of a federated link, then are you suggesting that this just
>> might be the cause of our woes?
>
> I'm not exactly sure where the problem is. Is it the 'upstream' broker
> that is growing? i.e. the one the publisher is connected to? Or is it
> the broker 'downstream' of the federation link (the one the consumer
> is connected to)?
It's the broker that the publisher is connected to that is growing. As I
mentioned previously the really weird thing is that it only seems to
happen when that broker is co-located with the publisher. What's
especially weird is that things are mostly OK. most of our topology
still has the producer and first broker co-located, but we had a
particularly high rate (and quite bursty) producer that was giving us
problems and the work around was to point it to a remote broker. I
suspect that we're seeing it again because many of our other producers
are starting to increase their rate now.
Our theory has been that perhaps the network was doing just enough to
flatten out the burstiness. The test clients are set up to deliver
bursts then sleep as this is quite close to what our real producers do.
>
>> At this stage having a
>> good analysis and understanding is probably most important - so if you
>> can help me enable flow control (and suggest how I'd try unreliable
>> messaging using qpid::client too if possible) in the client-consumer I
>> posted that would be really helpful.
>
> subscription = subscriptions.subscribe(*this, queue,
> SubscriptionSettings(FlowControl::messageWindow(100),
> ACCEPT_MODE_NONE));
>
> Would give a flow controlled, unreliable subscription. However simply
> setting the ACCEPT_MODE_NONE with FlowControl::unlimited() in the
> qpid::client API should prevent build up of messages through delivery
> records at all.
Thanks, I'll try that out.
>
>> If I can get evidence to suggest that the federation "unreliable links"
>> behaviour isn't as effective at mitigating this in 0.8 as it is in later
>> brokers that would be useful, as I say I keep arguing that we should be
>> upgrading to something newer.
>
> I've had a scan through the svn logs and I don't see anything
> obviously broken in 0.8; the lines that allow the message to be
> released are in there. However a lot has changed since and many other
> things could interact.
It's a pity that there's nothing obvious. I'll try your suggestion for
unreliable messaging then I might set my box back to 0.8 to see if it
behaves any differently than 0.12.
BTW do you know of a good way to allow multiple broker versions to
co-exist on the same box (short of virtualisation). I always seem to
need to completely uninstall one version before I can install another
due to library dependencies. I guess that I could set up a chroot jail,
but I was wondering if there's a simpler approach?
Frase.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: more memory "leak" woes.
Posted by Gordon Sim <gs...@redhat.com>.
On 08/03/2012 01:35 PM, Fraser Adams wrote:
> So if the "unreliable message" behaviour isn't behaving as expected in
> earlier brokers (say 0.8) and we have a slow consumer hanging off the
> other side of a federated link, then are you suggesting that this just
> might be the cause of our woes?
I'm not exactly sure where the problem is. Is it the 'upstream' broker
that is growing? i.e. the one the publisher is connected to? Or is it
the broker 'downstream' of the federation link (the one the consumer is
connected to)?
> At this stage having a
> good analysis and understanding is probably most important - so if you
> can help me enable flow control (and suggest how I'd try unreliable
> messaging using qpid::client too if possible) in the client-consumer I
> posted that would be really helpful.
subscription = subscriptions.subscribe(*this, queue,
SubscriptionSettings(FlowControl::messageWindow(100),
ACCEPT_MODE_NONE));
Would give a flow controlled, unreliable subscription. However simply
setting the ACCEPT_MODE_NONE with FlowControl::unlimited() in the
qpid::client API should prevent build up of messages through delivery
records at all.
> If I can get evidence to suggest that the federation "unreliable links"
> behaviour isn't as effective at mitigating this in 0.8 as it is in later
> brokers that would be useful, as I say I keep arguing that we should be
> upgrading to something newer.
I've had a scan through the svn logs and I don't see anything obviously
broken in 0.8; the lines that allow the message to be released are in
there. However a lot has changed since and many other things could interact.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: more memory "leak" woes.
Posted by Gordon Sim <gs...@redhat.com>.
On 08/03/2012 02:29 PM, Fraser Adams wrote:
> Any idea why calling setFlowControl() on the subscription didn't work?
Your initial subscription sets infinite credit. The subsequent call to
setFlowControl() adds credit. What is required is first to cancel the
existing infinite credit (through a messageStop()).
> It seems confusing to have a method call that doesn't appear to work...
I would agree its not intuitive. I suspect exposing that method was not
considered carefully enough.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: more memory "leak" woes.
Posted by Fraser Adams <fr...@blueyonder.co.uk>.
On 03/08/12 13:56, Gordon Sim wrote:
> On 08/03/2012 01:55 PM, Gordon Sim wrote:
>> On 08/03/2012 01:35 PM, Fraser Adams wrote:
>>> subscription = subscriptions.subscribe(*this, queue);
>>>
>>> subscription.setFlowControl(FlowControl::messageWindow(100)); // Why
>>> doesn't this work :-( ???
>>> subscription.setAutoAck(10);
>>
>> When you say it didn't work, what exactly do you mean? That it behaved
>> like the messaqeCredit() call? or that it didn't restrict the window?
>>
>> Try setting the option on subscribe, rather than after it. E.g.
>>
>> subscription = subscriptions.subscribe(*this, queue,
>> SubscriptionSettings(FlowControl::messageCredit(100)));
>
> Sorry! That should read messageWindow() of course!!
>
So yeah when I tried
subscription.setFlowControl(FlowControl::messageWindow(100));
It behaved as if I hadn't done anything and my memory consumption grew
out of control.
I've now tried
subscription = subscriptions.subscribe(*this, queue,
SubscriptionSettings(FlowControl::messageWindow(100)));
subscription.setAutoAck(10);
As you suggested here and both qpidd and client-consumer seem happier -
qpidd is on ~320M and client-consumer is on ~220M SHR according to top
and are both fairly stable.
Any idea why calling setFlowControl() on the subscription didn't work?
It seems confusing to have a method call that doesn't appear to work...
On my other questions - so now I've got flow control working can you
suggest how to configure a qpid::client consumer to use unreliable
messaging?
So if I can configure client-consumer with unlimited flow control and
unreliable messaging will that be a fair emulation of a broker bridge
with a slow consumer hanging off it. I might then be able to try it with
my 0.12 broker then a 0.8 broker and see if they behave differently - if
they do then that just *might* be our smoking gun - what do you think
(unless you already know the answer).
Cheers,
Frase
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: more memory "leak" woes.
Posted by Gordon Sim <gs...@redhat.com>.
On 08/03/2012 01:55 PM, Gordon Sim wrote:
> On 08/03/2012 01:35 PM, Fraser Adams wrote:
>> subscription = subscriptions.subscribe(*this, queue);
>>
>> subscription.setFlowControl(FlowControl::messageWindow(100)); // Why
>> doesn't this work :-( ???
>> subscription.setAutoAck(10);
>
> When you say it didn't work, what exactly do you mean? That it behaved
> like the messaqeCredit() call? or that it didn't restrict the window?
>
> Try setting the option on subscribe, rather than after it. E.g.
>
> subscription = subscriptions.subscribe(*this, queue,
> SubscriptionSettings(FlowControl::messageCredit(100)));
Sorry! That should read messageWindow() of course!!
> (or else try a Session::messageStop() passing in the subscription name
> before changing the flow control settings).
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: more memory "leak" woes.
Posted by Gordon Sim <gs...@redhat.com>.
On 08/03/2012 01:35 PM, Fraser Adams wrote:
> subscription = subscriptions.subscribe(*this, queue);
>
> subscription.setFlowControl(FlowControl::messageWindow(100)); // Why
> doesn't this work :-( ???
> subscription.setAutoAck(10);
When you say it didn't work, what exactly do you mean? That it behaved
like the messaqeCredit() call? or that it didn't restrict the window?
Try setting the option on subscribe, rather than after it. E.g.
subscription = subscriptions.subscribe(*this, queue,
SubscriptionSettings(FlowControl::messageCredit(100)));
(or else try a Session::messageStop() passing in the subscription name
before changing the flow control settings).
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: more memory "leak" woes.
Posted by Fraser Adams <fr...@blueyonder.co.uk>.
Thanks for the quick response Gordon!
>
> For a reliable receiver, the broker needs to keep track of all
> unacknowledged messages. That will include all prefetched messages
> (depending on the frequency at which acknowledgements are actually
> sent it may include fetched messages also of course).
We were beginning to wonder if that was what was happening. I guess
that's borne out by my experiments with the messaging-consumer capacity.
>
> If the receiver is unreliable (no explicit acknowledgement expected)
> then (at least in recent brokers) the record will no longer reference
> the message, but simply keep track of the amount of byte credit it
> consumed in order to move the window correctly.
when you say recent - what version? I've been testing at home with 0.12
and when I added the link unreliable stuff to my qpid::messaging client
the consumption seems stable, so I'm guessing 0.12 is OK. Unfortunately
at work the brokers are still 0.8 - I'm pushing to uplift, maybe this
will help my corner....
>
> Yes, you would need to use FlowControl::messageWindow() instead; the
> window will be moved automatically by the library unless you choose to
> handle completions manually. By default qpid::client subscriptions use
> unlimited credit (meaning the prefetch is unbounded).
Hmmm I tried swapping messageCredit() to messageWindow() to no avail -
see below
void Listener::prepareQueue(string queue, string exchange)
{
queue += session.getId().getName();
cout << "Declaring queue: " << queue << endl;
FieldTable q_fields;
q_fields.setString("qpid.policy_type", "ring");
q_fields.setUInt64("qpid.max_size", 100000000/*1000000000*/);
session.queueDeclare(arg::queue = queue, arg::exclusive = true,
arg::autoDelete = true, arg::arguments = q_fields);
FieldTable b_flds;
b_flds.setString("x-match", "all");
b_flds.setString("data-format", "test");
session.exchangeBind(arg::exchange = exchange, arg::queue = queue,
arg::arguments = b_flds);
cout << "Subscribing to queue " << queue << endl;
subscription = subscriptions.subscribe(*this, queue);
subscription.setFlowControl(FlowControl::messageWindow(100)); // Why
doesn't this work :-( ???
subscription.setAutoAck(10);
}
I'm really struggling with qpid::client flow control - I tried looking
through
https://svn.apache.org/repos/asf/qpid/trunk/qpid/cpp/src/tests/qpid-topic-listener.cpp
which I found Googling qpid::client flow control, but as I say when I
did the
SubscriptionSettings settings;
settings.autoAck = 10;
settings.flowControl = FlowControl::messageCredit(100);
listener.subscription = mgr.subscribe(listener, control, settings);
It just hung at 100 messages - clearly qpid::client doesn't like me! the
feeling is mutual :-)
>
> (Yes, I agree its not the easiest API to use and has a fair amount of
> incidental complexity, hence qpid::messaging!)
Your telling me :-)
>
> Indeed, if you are not using the --ack option then there are no
> acknowledgements required. Though the federation subscriptions do use
> infinite credit, this should at least prevent holding on to messages.
> What version of the broker are you using now? (I recall at one time
> there was an older broker in part of the system?)
So we're definitely not using the --ack option, but the " federation
subscriptions do use infinite credit" is troubling. As I say we're using
0.8 brokers.
So if the "unreliable message" behaviour isn't behaving as expected in
earlier brokers (say 0.8) and we have a slow consumer hanging off the
other side of a federated link, then are you suggesting that this just
might be the cause of our woes?
>
> I could probably get you a simple patch if applying that and
> rebuilding is an option.
>
It might be an option - thanks for the offer! At this stage having a
good analysis and understanding is probably most important - so if you
can help me enable flow control (and suggest how I'd try unreliable
messaging using qpid::client too if possible) in the client-consumer I
posted that would be really helpful.
If I can get evidence to suggest that the federation "unreliable links"
behaviour isn't as effective at mitigating this in 0.8 as it is in later
brokers that would be useful, as I say I keep arguing that we should be
upgrading to something newer.
Many thanks again for your help.
Frase
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: more memory "leak" woes.
Posted by Gordon Sim <gs...@redhat.com>.
On 08/03/2012 11:42 AM, Fraser Adams wrote:
> Hello all,
> I've previously posted with a problem whereby in a federated set up
> we've been seeing brokers eat memory. In that case what was weird was
> that when the producer client was pointed to a broker on the same box
> we'd start to see problems, but when it was pointed to a broker on a
> remote box it largely seemed to be stable. It drove us nuts, but we were
> able to park it for a while. We've recently started seeing it again
> where we really had to use a co-located broker.
>
> One thing that seems interesting is that it happens most when the
> consumer was in some way behaving slower than it should.
>
> We're publishing messages to amq.match and the queue the consumers
> receive from is a RING queue, so as far as I can see (intuitively at
> least!!) it shouldn't really matter if the consumer is slow - the
> circular queue should just overwrite the oldest messages.
>
> One of my colleagues wrote a couple of little programs using the
> qpid::client API to reproduce a bursty producer and a slow consumer and
> this seems to reproduce the problem - basically despite a circular queue
> both qpidd and client-consumer memory consumption grows and I start
> swapping madly on my 4G box despite the queue only being 100MB.
>
> I'm more familiar with qpid::messaging, so I tried to reproduce the
> problem using that, what's interesting is that my first attempt couldn't
> reproduce it - but of course the APIs behave differently. Then I started
> wondering about flow control/capacity.
>
> I usually do setCapacity(500) - for no other reason than I think that's
> what the default is for the JMS API. Now with that I was using more
> qpidd memory than I'd expect with a 100M circular queue, but I then
> reduced it to 100 then 10 and realised that the capacity (which I
> thought related to prefetch on the client) was affecting both client and
> qpidd memory consumption. I also noticed that doing "link: {reliability:
> unreliable}" helped.
For a reliable receiver, the broker needs to keep track of all
unacknowledged messages. That will include all prefetched messages
(depending on the frequency at which acknowledgements are actually sent
it may include fetched messages also of course).
In the current broker these records will include a reference to the
message which prevents it being deleted, even if in the case of a ring
queue it is overwritten.
If the receiver is unreliable (no explicit acknowledgement expected)
then (at least in recent brokers) the record will no longer reference
the message, but simply keep track of the amount of byte credit it
consumed in order to move the window correctly.
> I tried enabling flow control in the client-consumer to no avail (I
> struggle to understand that API!!) - I thought adding
>
> SubscriptionSettings settings;
> settings.autoAck = 100;
> settings.flowControl = FlowControl::messageCredit(200);
>
> subscription = subscriptions.subscribe(*this, queue, settings);
>
> to my prepareQueue() method would be the way to do it, but that just
> seemed to cause my consumer to hang after it had received 200 messages.
Yes, you would need to use FlowControl::messageWindow() instead; the
window will be moved automatically by the library unless you choose to
handle completions manually. By default qpid::client subscriptions use
unlimited credit (meaning the prefetch is unbounded).
(Yes, I agree its not the easiest API to use and has a fair amount of
incidental complexity, hence qpid::messaging!)
> So my messaging-consumer can be made to behave in a way that at least
> makes some sense, but what concerns me is back to the problem of
> federated links - as I say we're seeing terrible resource leaks and I
> assume that the federation bridge code is closer to what qpid::client is
> doing, so I've no idea if we can configure a federated link to "honour"
> the behaviour that we'd expect from a circular queue (we use the default
> behaviour which *should* we using unreliable messaging).
Indeed, if you are not using the --ack option then there are no
acknowledgements required. Though the federation subscriptions do use
infinite credit, this should at least prevent holding on to messages.
What version of the broker are you using now? (I recall at one time
there was an older broker in part of the system?)
> hope this makes sense. I've attached the producer and two consumers that
> I've been using to try this stuff out.
>
> I'd appreciate any thoughts and especially any mitigations, this is
> starting to cause us real problems.
I could probably get you a simple patch if applying that and rebuilding
is an option.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org