You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Shan Wang <Sh...@igindex.co.uk> on 2009/06/05 20:58:41 UTC
c++ client has memory leak when queue is full and the client tries to reconnect
When the queue reaches its full capacity, and a c++ client tries to put
a message to the queue, the client execute() will throw an exception and
the connection will be closed by broker. After that, if the client tries
to call FailoverManager::connect() again to the same queue, it will
succeed but the memory increases about 2MB each time it connects.
Sometimes my application still have tens of thousands of messages when
the queue is full and this problem has caused my program to crash after
exhausting all the available memory.
The queue is declared as durable, and the client program is not far from
the replay_sender example. I tried to close the connection explicitly
before reconnecting, or close/flush/suspend the session associated to
the connection, but in the end I still have the same result. Can anyone
please advice me how can I avoid this memory consumption? Or
alternatively, is there anyway to tell from the client side if the queue
it's talking to is full or not, so I don't have to call connect() until
the queue is cleared, of course assuming this method won't cause the
same memory leak.
Thanks a lot,
Shan
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44 (0)20 7896 0011) and then delete the email and any copies of it. Opinions, conclusions (etc.) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG Index plc is a company registered in England and Wales under number 01190902. VAT registration number 761 2978 07. Registered Office: Friars House, 157-168 Blackfriars Road, London SE1 8EZ. Authorised and regulated by the Financial Services Authority. FSA Register number 114059.
Re: c++ client has memory leak when queue is full and the client
tries to reconnect
Posted by Gordon Sim <gs...@redhat.com>.
Shan Wang wrote:
> The client is using 0.5 built from trunk in early April, so maybe
> different from the current 0.5 release version.
Hmm, thats odd. The fix was made on the 13th February (r744051) so you
should have it.
>
> -----Original Message-----
> From: Gordon Sim [mailto:gsim@redhat.com]
> Sent: 09 June 2009 13:37
> To: users@qpid.apache.org
> Cc: dev@qpid.apache.org
> Subject: Re: c++ client has memory leak when queue is full and the
> client tries to reconnect
>
> Shan Wang wrote:
>> I fixed it by adding session.close() before calling
>> FailoverManager::connect() again after disconnect. So it seems the
>> resource leaked was from uncleaned sessions. I forgot to mention this
>> problem only affects Windows client.
>
> Ok, glad you got it fixed though it sounds like there may be some fix
> also required in the windows client side IO(?).
>
>> Besides, on windows I can call
>> session.close() without having the session binding with any
> connection(
>> I mean creating a session object but haven't called
>> connect().newSession() ), but on Linux this will cause a crash, the
>> cause seems to be from the mutex the session is using.
>
> What version are you using? Invoking close on an invalid session (i.e.
> one not returned by newSession()) should no longer result in an error.
> Invoking other methods will do however.
>
>> Another question is, what the FailoverManager does when the current
>> connection becomes unavailable? As I understand it, FailoverManager
> will
>> try to reconnect once according to the policy specified and if failed
> it
>> will set the state to CAN'T CONNECT and give up. So if my cluster has
>> two brokers and both of them are out of reach, am I right that client
>> application has to reconnect by calling
>> FailoverManager::connect()explicitly?
>
> Yes, that is right.
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:dev-subscribe@qpid.apache.org
>
> The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44 (0)20 7896 0011) and then delete the email and any copies of it. Opinions, conclusions (etc.) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG Index plc is a company registered in England and Wales under number 01190902. VAT registration number 761 2978 07. Registered Office: Friars House, 157-168 Blackfriars Road, London SE1 8EZ. Authorised and regulated by the Financial Services Authority. FSA Register number 114059.
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:dev-subscribe@qpid.apache.org
>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
RE: c++ client has memory leak when queue is full and the client tries to reconnect
Posted by Shan Wang <Sh...@igindex.co.uk>.
The client is using 0.5 built from trunk in early April, so maybe
different from the current 0.5 release version.
-----Original Message-----
From: Gordon Sim [mailto:gsim@redhat.com]
Sent: 09 June 2009 13:37
To: users@qpid.apache.org
Cc: dev@qpid.apache.org
Subject: Re: c++ client has memory leak when queue is full and the
client tries to reconnect
Shan Wang wrote:
> I fixed it by adding session.close() before calling
> FailoverManager::connect() again after disconnect. So it seems the
> resource leaked was from uncleaned sessions. I forgot to mention this
> problem only affects Windows client.
Ok, glad you got it fixed though it sounds like there may be some fix
also required in the windows client side IO(?).
> Besides, on windows I can call
> session.close() without having the session binding with any
connection(
> I mean creating a session object but haven't called
> connect().newSession() ), but on Linux this will cause a crash, the
> cause seems to be from the mutex the session is using.
What version are you using? Invoking close on an invalid session (i.e.
one not returned by newSession()) should no longer result in an error.
Invoking other methods will do however.
> Another question is, what the FailoverManager does when the current
> connection becomes unavailable? As I understand it, FailoverManager
will
> try to reconnect once according to the policy specified and if failed
it
> will set the state to CAN'T CONNECT and give up. So if my cluster has
> two brokers and both of them are out of reach, am I right that client
> application has to reconnect by calling
> FailoverManager::connect()explicitly?
Yes, that is right.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44 (0)20 7896 0011) and then delete the email and any copies of it. Opinions, conclusions (etc.) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG Index plc is a company registered in England and Wales under number 01190902. VAT registration number 761 2978 07. Registered Office: Friars House, 157-168 Blackfriars Road, London SE1 8EZ. Authorised and regulated by the Financial Services Authority. FSA Register number 114059.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: c++ client has memory leak when queue is full and the client
tries to reconnect
Posted by Gordon Sim <gs...@redhat.com>.
Shan Wang wrote:
> I fixed it by adding session.close() before calling
> FailoverManager::connect() again after disconnect. So it seems the
> resource leaked was from uncleaned sessions. I forgot to mention this
> problem only affects Windows client.
Ok, glad you got it fixed though it sounds like there may be some fix
also required in the windows client side IO(?).
> Besides, on windows I can call
> session.close() without having the session binding with any connection(
> I mean creating a session object but haven't called
> connect().newSession() ), but on Linux this will cause a crash, the
> cause seems to be from the mutex the session is using.
What version are you using? Invoking close on an invalid session (i.e.
one not returned by newSession()) should no longer result in an error.
Invoking other methods will do however.
> Another question is, what the FailoverManager does when the current
> connection becomes unavailable? As I understand it, FailoverManager will
> try to reconnect once according to the policy specified and if failed it
> will set the state to CAN'T CONNECT and give up. So if my cluster has
> two brokers and both of them are out of reach, am I right that client
> application has to reconnect by calling
> FailoverManager::connect()explicitly?
Yes, that is right.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: c++ client has memory leak when queue is full and the client
tries to reconnect
Posted by Gordon Sim <gs...@redhat.com>.
Shan Wang wrote:
> I fixed it by adding session.close() before calling
> FailoverManager::connect() again after disconnect. So it seems the
> resource leaked was from uncleaned sessions. I forgot to mention this
> problem only affects Windows client.
Ok, glad you got it fixed though it sounds like there may be some fix
also required in the windows client side IO(?).
> Besides, on windows I can call
> session.close() without having the session binding with any connection(
> I mean creating a session object but haven't called
> connect().newSession() ), but on Linux this will cause a crash, the
> cause seems to be from the mutex the session is using.
What version are you using? Invoking close on an invalid session (i.e.
one not returned by newSession()) should no longer result in an error.
Invoking other methods will do however.
> Another question is, what the FailoverManager does when the current
> connection becomes unavailable? As I understand it, FailoverManager will
> try to reconnect once according to the policy specified and if failed it
> will set the state to CAN'T CONNECT and give up. So if my cluster has
> two brokers and both of them are out of reach, am I right that client
> application has to reconnect by calling
> FailoverManager::connect()explicitly?
Yes, that is right.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
RE: c++ client has memory leak when queue is full and the client tries to reconnect
Posted by Shan Wang <Sh...@igindex.co.uk>.
Hi,
Thanks for the reply.
I fixed it by adding session.close() before calling
FailoverManager::connect() again after disconnect. So it seems the
resource leaked was from uncleaned sessions. I forgot to mention this
problem only affects Windows client. Besides, on windows I can call
session.close() without having the session binding with any connection(
I mean creating a session object but haven't called
connect().newSession() ), but on Linux this will cause a crash, the
cause seems to be from the mutex the session is using.
Another question is, what the FailoverManager does when the current
connection becomes unavailable? As I understand it, FailoverManager will
try to reconnect once according to the policy specified and if failed it
will set the state to CAN'T CONNECT and give up. So if my cluster has
two brokers and both of them are out of reach, am I right that client
application has to reconnect by calling
FailoverManager::connect()explicitly?
-----Original Message-----
From: Gordon Sim [mailto:gsim@redhat.com]
Sent: 08 June 2009 13:05
To: dev@qpid.apache.org
Cc: users@qpid.apache.org
Subject: Re: c++ client has memory leak when queue is full and the
client tries to reconnect
Shan Wang wrote:
> When the queue reaches its full capacity, and a c++ client tries to
put
> a message to the queue, the client execute() will throw an exception
and
> the connection will be closed by broker. After that, if the client
tries
> to call FailoverManager::connect() again to the same queue, it will
> succeed but the memory increases about 2MB each time it connects.
> Sometimes my application still have tens of thousands of messages when
> the queue is full and this problem has caused my program to crash
after
> exhausting all the available memory.
>
>
>
> The queue is declared as durable, and the client program is not far
from
> the replay_sender example. I tried to close the connection explicitly
> before reconnecting, or close/flush/suspend the session associated to
> the connection, but in the end I still have the same result. Can
anyone
> please advice me how can I avoid this memory consumption?
I suspect that you are queueing up messages in the MessageReplayTracker
that are from old sessions and thus will never be cleared. The isRetry
flag on FailoverManager::Command::execute() will only be set if the
FailoverManager itself re-executes the command due to a transport
failure.
If you are explicitly re-executing the command after an explicit
application level error, then the FailoverManager will treat it is a new
command. Consequently if you are using the isRetry flag to determine
whether to replay messages or not (as the replaying _sender example
does), then the replay will not get called. This leaves messages from
the old session in the buffer and these won't be cleared as the command
ids will not match the completion information from the current session.
It will also mean that you have gaps in the sequence of messages (which
you are presumably wanting to avoid as you are using
MessageReplayTracker in the first place).
I hope the above explanation helps. The simple solution to your problem
(assuming I have diagnosed it correctly) is to replace:
if (isRetry) sender.replay(session);
else sender.init(session);
with just:
sender.replay(session);
> Or
> alternatively, is there anyway to tell from the client side if the
queue
> it's talking to is full or not, so I don't have to call connect()
until
> the queue is cleared, of course assuming this method won't cause the
> same memory leak.
The exception thrown gives an indication of why the connection/session
was aborted.
When a configured limit on the in-memory queue is reached and the policy
is to reject sessions that try to publish more messages for example the
exception will be qpid::framing::ResourceLimitExceededException.
--Gordon.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44 (0)20 7896 0011) and then delete the email and any copies of it. Opinions, conclusions (etc.) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG Index plc is a company registered in England and Wales under number 01190902. VAT registration number 761 2978 07. Registered Office: Friars House, 157-168 Blackfriars Road, London SE1 8EZ. Authorised and regulated by the Financial Services Authority. FSA Register number 114059.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
RE: c++ client has memory leak when queue is full and the client tries to reconnect
Posted by Shan Wang <Sh...@igindex.co.uk>.
Hi,
Thanks for the reply.
I fixed it by adding session.close() before calling
FailoverManager::connect() again after disconnect. So it seems the
resource leaked was from uncleaned sessions. I forgot to mention this
problem only affects Windows client. Besides, on windows I can call
session.close() without having the session binding with any connection(
I mean creating a session object but haven't called
connect().newSession() ), but on Linux this will cause a crash, the
cause seems to be from the mutex the session is using.
Another question is, what the FailoverManager does when the current
connection becomes unavailable? As I understand it, FailoverManager will
try to reconnect once according to the policy specified and if failed it
will set the state to CAN'T CONNECT and give up. So if my cluster has
two brokers and both of them are out of reach, am I right that client
application has to reconnect by calling
FailoverManager::connect()explicitly?
-----Original Message-----
From: Gordon Sim [mailto:gsim@redhat.com]
Sent: 08 June 2009 13:05
To: dev@qpid.apache.org
Cc: users@qpid.apache.org
Subject: Re: c++ client has memory leak when queue is full and the
client tries to reconnect
Shan Wang wrote:
> When the queue reaches its full capacity, and a c++ client tries to
put
> a message to the queue, the client execute() will throw an exception
and
> the connection will be closed by broker. After that, if the client
tries
> to call FailoverManager::connect() again to the same queue, it will
> succeed but the memory increases about 2MB each time it connects.
> Sometimes my application still have tens of thousands of messages when
> the queue is full and this problem has caused my program to crash
after
> exhausting all the available memory.
>
>
>
> The queue is declared as durable, and the client program is not far
from
> the replay_sender example. I tried to close the connection explicitly
> before reconnecting, or close/flush/suspend the session associated to
> the connection, but in the end I still have the same result. Can
anyone
> please advice me how can I avoid this memory consumption?
I suspect that you are queueing up messages in the MessageReplayTracker
that are from old sessions and thus will never be cleared. The isRetry
flag on FailoverManager::Command::execute() will only be set if the
FailoverManager itself re-executes the command due to a transport
failure.
If you are explicitly re-executing the command after an explicit
application level error, then the FailoverManager will treat it is a new
command. Consequently if you are using the isRetry flag to determine
whether to replay messages or not (as the replaying _sender example
does), then the replay will not get called. This leaves messages from
the old session in the buffer and these won't be cleared as the command
ids will not match the completion information from the current session.
It will also mean that you have gaps in the sequence of messages (which
you are presumably wanting to avoid as you are using
MessageReplayTracker in the first place).
I hope the above explanation helps. The simple solution to your problem
(assuming I have diagnosed it correctly) is to replace:
if (isRetry) sender.replay(session);
else sender.init(session);
with just:
sender.replay(session);
> Or
> alternatively, is there anyway to tell from the client side if the
queue
> it's talking to is full or not, so I don't have to call connect()
until
> the queue is cleared, of course assuming this method won't cause the
> same memory leak.
The exception thrown gives an indication of why the connection/session
was aborted.
When a configured limit on the in-memory queue is reached and the policy
is to reject sessions that try to publish more messages for example the
exception will be qpid::framing::ResourceLimitExceededException.
--Gordon.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44 (0)20 7896 0011) and then delete the email and any copies of it. Opinions, conclusions (etc.) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG Index plc is a company registered in England and Wales under number 01190902. VAT registration number 761 2978 07. Registered Office: Friars House, 157-168 Blackfriars Road, London SE1 8EZ. Authorised and regulated by the Financial Services Authority. FSA Register number 114059.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: c++ client has memory leak when queue is full and the client
tries to reconnect
Posted by Gordon Sim <gs...@redhat.com>.
Shan Wang wrote:
> When the queue reaches its full capacity, and a c++ client tries to put
> a message to the queue, the client execute() will throw an exception and
> the connection will be closed by broker. After that, if the client tries
> to call FailoverManager::connect() again to the same queue, it will
> succeed but the memory increases about 2MB each time it connects.
> Sometimes my application still have tens of thousands of messages when
> the queue is full and this problem has caused my program to crash after
> exhausting all the available memory.
>
>
>
> The queue is declared as durable, and the client program is not far from
> the replay_sender example. I tried to close the connection explicitly
> before reconnecting, or close/flush/suspend the session associated to
> the connection, but in the end I still have the same result. Can anyone
> please advice me how can I avoid this memory consumption?
I suspect that you are queueing up messages in the MessageReplayTracker
that are from old sessions and thus will never be cleared. The isRetry
flag on FailoverManager::Command::execute() will only be set if the
FailoverManager itself re-executes the command due to a transport failure.
If you are explicitly re-executing the command after an explicit
application level error, then the FailoverManager will treat it is a new
command. Consequently if you are using the isRetry flag to determine
whether to replay messages or not (as the replaying _sender example
does), then the replay will not get called. This leaves messages from
the old session in the buffer and these won't be cleared as the command
ids will not match the completion information from the current session.
It will also mean that you have gaps in the sequence of messages (which
you are presumably wanting to avoid as you are using
MessageReplayTracker in the first place).
I hope the above explanation helps. The simple solution to your problem
(assuming I have diagnosed it correctly) is to replace:
if (isRetry) sender.replay(session);
else sender.init(session);
with just:
sender.replay(session);
> Or
> alternatively, is there anyway to tell from the client side if the queue
> it's talking to is full or not, so I don't have to call connect() until
> the queue is cleared, of course assuming this method won't cause the
> same memory leak.
The exception thrown gives an indication of why the connection/session
was aborted.
When a configured limit on the in-memory queue is reached and the policy
is to reject sessions that try to publish more messages for example the
exception will be qpid::framing::ResourceLimitExceededException.
--Gordon.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: c++ client has memory leak when queue is full and the client
tries to reconnect
Posted by Gordon Sim <gs...@redhat.com>.
Shan Wang wrote:
> When the queue reaches its full capacity, and a c++ client tries to put
> a message to the queue, the client execute() will throw an exception and
> the connection will be closed by broker. After that, if the client tries
> to call FailoverManager::connect() again to the same queue, it will
> succeed but the memory increases about 2MB each time it connects.
> Sometimes my application still have tens of thousands of messages when
> the queue is full and this problem has caused my program to crash after
> exhausting all the available memory.
>
>
>
> The queue is declared as durable, and the client program is not far from
> the replay_sender example. I tried to close the connection explicitly
> before reconnecting, or close/flush/suspend the session associated to
> the connection, but in the end I still have the same result. Can anyone
> please advice me how can I avoid this memory consumption?
I suspect that you are queueing up messages in the MessageReplayTracker
that are from old sessions and thus will never be cleared. The isRetry
flag on FailoverManager::Command::execute() will only be set if the
FailoverManager itself re-executes the command due to a transport failure.
If you are explicitly re-executing the command after an explicit
application level error, then the FailoverManager will treat it is a new
command. Consequently if you are using the isRetry flag to determine
whether to replay messages or not (as the replaying _sender example
does), then the replay will not get called. This leaves messages from
the old session in the buffer and these won't be cleared as the command
ids will not match the completion information from the current session.
It will also mean that you have gaps in the sequence of messages (which
you are presumably wanting to avoid as you are using
MessageReplayTracker in the first place).
I hope the above explanation helps. The simple solution to your problem
(assuming I have diagnosed it correctly) is to replace:
if (isRetry) sender.replay(session);
else sender.init(session);
with just:
sender.replay(session);
> Or
> alternatively, is there anyway to tell from the client side if the queue
> it's talking to is full or not, so I don't have to call connect() until
> the queue is cleared, of course assuming this method won't cause the
> same memory leak.
The exception thrown gives an indication of why the connection/session
was aborted.
When a configured limit on the in-memory queue is reached and the policy
is to reject sessions that try to publish more messages for example the
exception will be qpid::framing::ResourceLimitExceededException.
--Gordon.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org