You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Shan Wang <Sh...@igindex.co.uk> on 2009/03/16 20:09:10 UTC
a question about the FailoverManager
Hi All,
My questions are 1.) how fast can the FailoverManager handle a failover?
and 2.) will FailoverManager handle the case that all the brokers are
unavailable?
As I understand it, FailoverManager::execute() will try to reconnect if
no open connection is available, by using the last known brokers
list(because it uses connect() which has an empty brokers url list?),
and if that succeeded, a new session will be created and
Command::execute() will be called. And if the reconnect failed, it will
through an exception.
If the above is correct, then when broker1 in my cluster is dead, the
messages sent by sender should immediately failover to broker2, and the
client shouldn't notice anything. But in my test, what actually happened
is it took about 6-7 minutes for the clients to failover, during that
time both the sender and receiver clients were frozen.
And if I kill both of the brokers in the cluster, it also takes about
6-7 minutes for the clients to notice that there are no connections
available. Even if I restart the brokers within that frozen time,
clients still won't be able to connect the newly started broker but will
fail with "Cannot establish a connection" exception at last.
My test clients are almost the same as replay_sender and resume_receiver
examples from M4, the only variation is in the beginning they use
connect( url ) to connect the cluster rather than a specific host:port
string. Also I put sleep between each send so I can manage the test
easier.
So am I doing something wrong here? Can anyone suggest what do I need to
do in my application to have maximum resilience besides relying on
FailoverManager?
Thanks & Regards,
Shan
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44 (0)20 7896 0011) and then delete the email and any copies of it. Opinions, conclusions (etc.) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG Index plc is a company registered in England and Wales under number 01190902. VAT registration number 761 2978 07. Registered Office: Friars House, 157-168 Blackfriars Road, London SE1 8EZ. Authorised and regulated by the Financial Services Authority. FSA Register number 114059.
RE: a question about the FailoverManager
Posted by Shan Wang <Sh...@igindex.co.uk>.
Hi Gordon,
The original replay_sender does have the same problem, I killed the
broker by using kill pid. Pstack showed the following when replay_sender
is frozen:
#0 0xffffe410 in __kernel_vsyscall ()
#1 0x00586748 in connect () from /lib/tls/libc.so.6
#2 0x002809a1 in qpid::sys::Socket::connect ()
#3 0x00b89b75 in qpid::client::TCPConnector::connect ()
#4 0x00b79c24 in qpid::client::ConnectionImpl::open ()
#5 0x00b6a022 in qpid::client::Connection::open ()
#6 0x00ba7d5d in qpid::client::FailoverManager::attempt ()
#7 0x00ba9372 in qpid::client::FailoverManager::attempt ()
#8 0x00ba9abb in qpid::client::FailoverManager::connect ()
#9 0x00baac6b in qpid::client::FailoverManager::execute ()
Thanks,
Shan
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44 (0)20 7896 0011) and then delete the email and any copies of it. Opinions, conclusions (etc.) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG Index plc is a company registered in England and Wales under number 01190902. VAT registration number 761 2978 07. Registered Office: Friars House, 157-168 Blackfriars Road, London SE1 8EZ. Authorised and regulated by the Financial Services Authority. FSA Register number 114059.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: a question about the FailoverManager
Posted by Gordon Sim <gs...@redhat.com>.
Shan Wang wrote:
> If the above is correct, then when broker1 in my cluster is dead, the
> messages sent by sender should immediately failover to broker2, and the
> client shouldn't notice anything. But in my test, what actually happened
> is it took about 6-7 minutes for the clients to failover, during that
> time both the sender and receiver clients were frozen.
What does pstack report for the client processes when they are in that
'frozen' state?
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: a question about the FailoverManager
Posted by Carl Trieloff <cc...@redhat.com>.
>
> Yes, that is something to warn people about. Probably a FAQ item or
> something.
>
Added to the FAQ.
http://qpid.apache.org/faq.html
should sync in a bit
Carl.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: a question about the FailoverManager
Posted by Gordon Sim <gs...@redhat.com>.
Shan Wang wrote:
> It finally worked!!!
Wonderful!
> The problem is like you said, the published url list from brokers. When
> I use cluster-url option to publish only one interface, the failover
> became very quick, almost unnoticeable.
>
> I guess the reason is the other two interfaces on each broker are
> unreachable from the host I run my clients, so it took a long time for
> the socket to timeout until it got to the one worked, which is the last
> in the list.
Yes, that is something to warn people about. Probably a FAQ item or
something.
> Thanks for your help.
Thanks very much for your patience and perseverance!
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
RE: a question about the FailoverManager
Posted by Shan Wang <Sh...@igindex.co.uk>.
It finally worked!!!
The problem is like you said, the published url list from brokers. When
I use cluster-url option to publish only one interface, the failover
became very quick, almost unnoticeable.
I guess the reason is the other two interfaces on each broker are
unreachable from the host I run my clients, so it took a long time for
the socket to timeout until it got to the one worked, which is the last
in the list.
Thanks for your help.
Regards,
Shan
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44 (0)20 7896 0011) and then delete the email and any copies of it. Opinions, conclusions (etc.) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG Index plc is a company registered in England and Wales under number 01190902. VAT registration number 761 2978 07. Registered Office: Friars House, 157-168 Blackfriars Road, London SE1 8EZ. Authorised and regulated by the Financial Services Authority. FSA Register number 114059.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: a question about the FailoverManager
Posted by Gordon Sim <gs...@redhat.com>.
Shan Wang wrote:
> I tried to set heartbeat to 5, 10, and 30 seconds, but it didn't seem to
> take any effect, the failover still takes ages. Actually in
> ConnectionSettings.h I found the following comments:
>
> /**
> * Allows a heartbeat frequency to be specified (this feature is
> * not yet implemented).
> */
> uint16_t heartbeat;
>
> I'm using the redhat build which I believe is based on M4 version, and
> the above code is from M4 source code too. I can see in trunk this has
> been fully implemented now, but I expect for such a important feature
> redhat should have had it in there release.
No, heartbeat support was not part of M4. (The mechanism for requesting
it was which is why you see the code above).
However I think there is something else going on here. In your pstack
output the client is clearly waiting for the socket to connect. The
heartbeat as currently implemented won't help with that (it only takes
effect after the connection has been established - in response to the
Connection.Open control sent by the broker in fact).
By default he failover uses the list of known URLs and this includes
retrying the original broker. However you are simulating failure, it is
resulting in the socket connection attempt waiting for a long time.
To avoid that you can try the approach shown in the resuming_receiver
example. I.e. implement FailoverManager::ReconnectionStrategy and rotate
the original list (or indeed pop the first item off), so that you try
the connect to the second broker.
(Note: you may also need to use the --cluster-url option to the broker
to prevent each node advertising itself on more than one interface in
which case each interface will be tried in turn).
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
RE: a question about the FailoverManager
Posted by Shan Wang <Sh...@igindex.co.uk>.
I tried to set heartbeat to 5, 10, and 30 seconds, but it didn't seem to
take any effect, the failover still takes ages. Actually in
ConnectionSettings.h I found the following comments:
/**
* Allows a heartbeat frequency to be specified (this feature is
* not yet implemented).
*/
uint16_t heartbeat;
I'm using the redhat build which I believe is based on M4 version, and
the above code is from M4 source code too. I can see in trunk this has
been fully implemented now, but I expect for such a important feature
redhat should have had it in there release.
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44 (0)20 7896 0011) and then delete the email and any copies of it. Opinions, conclusions (etc.) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG Index plc is a company registered in England and Wales under number 01190902. VAT registration number 761 2978 07. Registered Office: Friars House, 157-168 Blackfriars Road, London SE1 8EZ. Authorised and regulated by the Financial Services Authority. FSA Register number 114059.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: a question about the FailoverManager
Posted by Gordon Sim <gs...@redhat.com>.
Shan Wang wrote:
> Probably this is the reason. I use the following command to start my
> cluster:
>
> sudo /usr/sbin/qpidd -p 5673 -t --cluster-name=mycluster --store-dir
> /export/home/amqmadm/qpidclus --auth no --ssl-cert-password-file
> /export/home/amqmadm/sandbox/cert.password --ssl-cert-db
> /export/home/amqmadm/sandbox/test_cert_db
>
> The /etc/qpidd.conf file is empty, so I guess there's no heartbeat
> setup? How can I set the heartbeat in config file or through command
> line? I can't find any clues in the documents.
You set the heartbeat through the client on a per-connection basis. The
ConnectionSettings class has a heartbeat field indicating the interval
(in seconds) at which heartbeats should be sent. You can pass an
instance of this class to Connection::open().
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
RE: a question about the FailoverManager
Posted by Shan Wang <Sh...@igindex.co.uk>.
Probably this is the reason. I use the following command to start my
cluster:
sudo /usr/sbin/qpidd -p 5673 -t --cluster-name=mycluster --store-dir
/export/home/amqmadm/qpidclus --auth no --ssl-cert-password-file
/export/home/amqmadm/sandbox/cert.password --ssl-cert-db
/export/home/amqmadm/sandbox/test_cert_db
The /etc/qpidd.conf file is empty, so I guess there's no heartbeat
setup? How can I set the heartbeat in config file or through command
line? I can't find any clues in the documents.
Thanks,
Shan
-----Original Message-----
From: Carl Trieloff [mailto:cctrieloff@redhat.com]
Sent: 16 March 2009 20:49
To: users@qpid.apache.org
Subject: Re: a question about the FailoverManager
Gordon Sim wrote:
> Shan Wang wrote:
>> As I understand it, FailoverManager::execute() will try to reconnect
if
>> no open connection is available, by using the last known brokers
>> list(because it uses connect() which has an empty brokers url list?),
>> and if that succeeded, a new session will be created and
>> Command::execute() will be called. And if the reconnect failed, it
will
>> through an exception.
>
> That is correct; when the current connection fails it will try to
> reconnect using the list of known urls retrieved when connected. It
> will try each of these in turn then fail if it can't connect to any of
> them.
Another thing is to make sure you have configured a heartbeat, so the
fail-over happens on
heartbeat interval and not on socket timeout.
regards
Carl.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44 (0)20 7896 0011) and then delete the email and any copies of it. Opinions, conclusions (etc.) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG Index plc is a company registered in England and Wales under number 01190902. VAT registration number 761 2978 07. Registered Office: Friars House, 157-168 Blackfriars Road, London SE1 8EZ. Authorised and regulated by the Financial Services Authority. FSA Register number 114059.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: a question about the FailoverManager
Posted by Carl Trieloff <cc...@redhat.com>.
Gordon Sim wrote:
> Shan Wang wrote:
>> As I understand it, FailoverManager::execute() will try to reconnect if
>> no open connection is available, by using the last known brokers
>> list(because it uses connect() which has an empty brokers url list?),
>> and if that succeeded, a new session will be created and
>> Command::execute() will be called. And if the reconnect failed, it will
>> through an exception.
>
> That is correct; when the current connection fails it will try to
> reconnect using the list of known urls retrieved when connected. It
> will try each of these in turn then fail if it can't connect to any of
> them.
Another thing is to make sure you have configured a heartbeat, so the
fail-over happens on
heartbeat interval and not on socket timeout.
regards
Carl.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: a question about the FailoverManager
Posted by Gordon Sim <gs...@redhat.com>.
Shan Wang wrote:
> As I understand it, FailoverManager::execute() will try to reconnect if
> no open connection is available, by using the last known brokers
> list(because it uses connect() which has an empty brokers url list?),
> and if that succeeded, a new session will be created and
> Command::execute() will be called. And if the reconnect failed, it will
> through an exception.
That is correct; when the current connection fails it will try to
reconnect using the list of known urls retrieved when connected. It will
try each of these in turn then fail if it can't connect to any of them.
> If the above is correct, then when broker1 in my cluster is dead, the
> messages sent by sender should immediately failover to broker2, and the
> client shouldn't notice anything. But in my test, what actually happened
> is it took about 6-7 minutes for the clients to failover, during that
> time both the sender and receiver clients were frozen.
That is much slower than I would expect from my testing. If I kill -9 a
broker, failover takes no more than a second or so. Are you killing the
broker in your simulations? or turning off the machine? Or something else?
> And if I kill both of the brokers in the cluster, it also takes about
> 6-7 minutes for the clients to notice that there are no connections
> available. Even if I restart the brokers within that frozen time,
> clients still won't be able to connect the newly started broker but will
> fail with "Cannot establish a connection" exception at last.
>
> My test clients are almost the same as replay_sender and resume_receiver
> examples from M4, the only variation is in the beginning they use
> connect( url ) to connect the cluster rather than a specific host:port
> string. Also I put sleep between each send so I can manage the test
> easier.
Do you see the same slowness in failover with those original examples?
> So am I doing something wrong here? Can anyone suggest what do I need to
> do in my application to have maximum resilience besides relying on
> FailoverManager?
I would certainly expect to see _much_ faster failover than that. I'm
not sure what is causing the delay - you could try turning on heartbeats
in case it is somehow slow to detect the closed socket.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org