You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by "Lance D." <la...@gmail.com> on 2013/05/03 14:14:23 UTC
Single threaded Client
Hello,
I've got a question about Socket timeouts. I've got a program that must be
single threaded. I know that's got its own share of problems, but the
biggest one that I have right now is the case of trying to connect to a
broker on a host that doesn't exist.
I'll start with my configuration. I have a client running with the 0.14
API on a Redhat Linux OS. My program uses a lookup service to find brokers
and exchanges that are providing data. The client connects to each
exchange, pulls the data and disconnects. I don't need to do this
extremely quickly, but it really shouldn't take more than 1-2 seconds per
connection because they are all on a small network.
My problems begin when the lookup service gives me a broker address for a
host that is powered off. When I call open on the connection, the SYN
packet is sent and I'm stuck waiting for the kernel socket timeout to
expire before qpid throws that exception. That means that I'm stalled for
20 seconds for each down host.
So, my question to this wonderful group of users is this: what is the best
way (other than going multi-threaded), to reduce my wait from 20 seconds
down to 2 seconds?
Thanks all for the help!
-Lance
Re: Single threaded Client
Posted by "Lance D." <la...@gmail.com>.
Oops. I guess I did miss that one detail. I'm using the c++ API; sorry I
didn't mention that earlier.
>From what I could find in the source code, I tend to agree with Frase's
assessment. I even contemplated extending the ConnectionImpl class and
getting setting the non-blocking to true or tweaking the socket
parameters. It's still not out of the question, but obviously its not
ideal.
As for the lookup service, I'm actually using LDAP, which tells me host,
port, and exchange. I can't really modify it to do any sort of host
checking; however, I've contemplated having a multi-threaded app that just
checks to see of LDAP agrees with the broker (using QMF).
Thanks Jason and Frase. I appreciate the responses.
-Lance
On Fri, May 3, 2013 at 1:00 PM, Fraser Adams
<fr...@blueyonder.co.uk>wrote:
> Hi Lance,
> perhaps someone else might be able to give you a better answer, but from
> my recollection none of the APIs give any particularly easy way to retrieve
> the underlying socket. Indeed from memory any methods that do exist at that
> level have been private or protected scope. Another person was trying to
> get socket info a bit back so I took a bit of a look then and couldn't find
> anything useful.
>
> I suspect that your best bet might be to do your own "network probe" - you
> say that you are using a lookup service, is that your own or something off
> the shelf? As I say you could probably do a port scan for port 5672 and use
> success there to feed your lookup service.
>
> Sorry that I can't give a more useful response, but I suspect that's what
> I'd do if faced with a similar problem.
>
> I guess that you *might* be able to do something clever with select/epoll
> so if you get some info from your lookup service you could try to establish
> a raw socket connection and use select/poll to check for the available
> active file descriptors then you could close them and use the knowledge to
> open up real AMQP connections to the active hosts. At least that'll avoid
> you needing to use threads.
>
> <caveat>It's been a while since I did anything that low level, so I could
> be talking rubbish :-D </caveat>
>
> Frase
>
>
>
> On 03/05/13 13:14, Lance D. wrote:
>
>> Hello,
>>
>> I've got a question about Socket timeouts. I've got a program that must be
>> single threaded. I know that's got its own share of problems, but the
>> biggest one that I have right now is the case of trying to connect to a
>> broker on a host that doesn't exist.
>>
>> I'll start with my configuration. I have a client running with the 0.14
>> API on a Redhat Linux OS. My program uses a lookup service to find
>> brokers
>> and exchanges that are providing data. The client connects to each
>> exchange, pulls the data and disconnects. I don't need to do this
>> extremely quickly, but it really shouldn't take more than 1-2 seconds per
>> connection because they are all on a small network.
>>
>> My problems begin when the lookup service gives me a broker address for a
>> host that is powered off. When I call open on the connection, the SYN
>> packet is sent and I'm stuck waiting for the kernel socket timeout to
>> expire before qpid throws that exception. That means that I'm stalled for
>> 20 seconds for each down host.
>>
>> So, my question to this wonderful group of users is this: what is the best
>> way (other than going multi-threaded), to reduce my wait from 20 seconds
>> down to 2 seconds?
>>
>> Thanks all for the help!
>> -Lance
>>
>>
>
> ------------------------------**------------------------------**---------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.**org<us...@qpid.apache.org>
> For additional commands, e-mail: users-help@qpid.apache.org
>
>
Re: Single threaded Client
Posted by Fraser Adams <fr...@blueyonder.co.uk>.
Hi Lance,
perhaps someone else might be able to give you a better answer, but from
my recollection none of the APIs give any particularly easy way to
retrieve the underlying socket. Indeed from memory any methods that do
exist at that level have been private or protected scope. Another person
was trying to get socket info a bit back so I took a bit of a look then
and couldn't find anything useful.
I suspect that your best bet might be to do your own "network probe" -
you say that you are using a lookup service, is that your own or
something off the shelf? As I say you could probably do a port scan for
port 5672 and use success there to feed your lookup service.
Sorry that I can't give a more useful response, but I suspect that's
what I'd do if faced with a similar problem.
I guess that you *might* be able to do something clever with
select/epoll so if you get some info from your lookup service you could
try to establish a raw socket connection and use select/poll to check
for the available active file descriptors then you could close them and
use the knowledge to open up real AMQP connections to the active hosts.
At least that'll avoid you needing to use threads.
<caveat>It's been a while since I did anything that low level, so I
could be talking rubbish :-D </caveat>
Frase
On 03/05/13 13:14, Lance D. wrote:
> Hello,
>
> I've got a question about Socket timeouts. I've got a program that must be
> single threaded. I know that's got its own share of problems, but the
> biggest one that I have right now is the case of trying to connect to a
> broker on a host that doesn't exist.
>
> I'll start with my configuration. I have a client running with the 0.14
> API on a Redhat Linux OS. My program uses a lookup service to find brokers
> and exchanges that are providing data. The client connects to each
> exchange, pulls the data and disconnects. I don't need to do this
> extremely quickly, but it really shouldn't take more than 1-2 seconds per
> connection because they are all on a small network.
>
> My problems begin when the lookup service gives me a broker address for a
> host that is powered off. When I call open on the connection, the SYN
> packet is sent and I'm stuck waiting for the kernel socket timeout to
> expire before qpid throws that exception. That means that I'm stalled for
> 20 seconds for each down host.
>
> So, my question to this wonderful group of users is this: what is the best
> way (other than going multi-threaded), to reduce my wait from 20 seconds
> down to 2 seconds?
>
> Thanks all for the help!
> -Lance
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Single threaded Client
Posted by Jason Barto <ja...@gmail.com>.
Lance what language and client library are you using to connect? You'll
need to get a reference to the socket being used by the client library and
modify it's timeout limit.
Sincerely,
Jason
On May 3, 2013 1:15 PM, "Lance D." <la...@gmail.com> wrote:
> Hello,
>
> I've got a question about Socket timeouts. I've got a program that must be
> single threaded. I know that's got its own share of problems, but the
> biggest one that I have right now is the case of trying to connect to a
> broker on a host that doesn't exist.
>
> I'll start with my configuration. I have a client running with the 0.14
> API on a Redhat Linux OS. My program uses a lookup service to find brokers
> and exchanges that are providing data. The client connects to each
> exchange, pulls the data and disconnects. I don't need to do this
> extremely quickly, but it really shouldn't take more than 1-2 seconds per
> connection because they are all on a small network.
>
> My problems begin when the lookup service gives me a broker address for a
> host that is powered off. When I call open on the connection, the SYN
> packet is sent and I'm stuck waiting for the kernel socket timeout to
> expire before qpid throws that exception. That means that I'm stalled for
> 20 seconds for each down host.
>
> So, my question to this wonderful group of users is this: what is the best
> way (other than going multi-threaded), to reduce my wait from 20 seconds
> down to 2 seconds?
>
> Thanks all for the help!
> -Lance
>
Re: Single threaded Client
Posted by "Lance D." <la...@gmail.com>.
Honestly, I'm not going to make a case one way or the other. As a general
rule, I like my middleware to divorce me, the user, from the underlining
implementation. That said, I also want more control over how long the
middleware takes to do what I asked for.
As an alternative to exposing OS-provided timeouts or some other timer
functionality, would adding a non-blocking/passive open() call be
reasonable and plausible for the C++ client?
Thanks again?
-Lance
On Wed, May 8, 2013 at 2:17 PM, Andrew Stitcher <as...@redhat.com>wrote:
> On Mon, 2013-05-06 at 20:26 -0400, Lance D. wrote:
> > Thanks all for that advice. You all are exploring the same paths that
> I've
> > explored in some manner. I've got requirements that force my code to
> have
> > traceable paths, which makes multi-threading within the application
> > difficult. Right now, the option I like the best is probably the
> heartbeat
> > option solution.
>
> Note that the heartbeat solution is the way that you are supposed to do
> this, although it may not be adequately documented.
>
> >
> > On a side note, what are the chances that we could update the API to
> expose
> > the underlying socket and/or provide the ability to tweak the socket
> > options. The API already exposes the Nagle option, so I don't see that
> it
> > would be that blasphemous to add in other, similar options.
>
> There is a low chance you could persuade me it would be a good idea to
> do this! I might be persuaded that doing the opposite makes some sense
> though (in other words importing an existing socket and using that).
>
> The reason for this is that the underlying code uses an abstraction over
> sockets and that is purely internal - there may not always be a socket
> to expose and trying to do so would just make things hard for not a huge
> gain in my opinion. However importing an existing socket would work
> within the existing code structure to some extent.
>
> BTW exposing the tcp-nodelay setting was intended to allow the user to
> trade off latency against throughput, rather than to expose a socket
> option specifically. I originally suggested a naming along those lines,
> but tcp-nodelay stuck.
>
> In the tests we've run, we found that tcp-nodelay is the better setting
> in nearly every circumstance. So we've now made that the default setting
> for the C++ code on trunk.
>
> Andrew
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>
Re: Single threaded Client
Posted by Andrew Stitcher <as...@redhat.com>.
On Mon, 2013-05-06 at 20:26 -0400, Lance D. wrote:
> Thanks all for that advice. You all are exploring the same paths that I've
> explored in some manner. I've got requirements that force my code to have
> traceable paths, which makes multi-threading within the application
> difficult. Right now, the option I like the best is probably the heartbeat
> option solution.
Note that the heartbeat solution is the way that you are supposed to do
this, although it may not be adequately documented.
>
> On a side note, what are the chances that we could update the API to expose
> the underlying socket and/or provide the ability to tweak the socket
> options. The API already exposes the Nagle option, so I don't see that it
> would be that blasphemous to add in other, similar options.
There is a low chance you could persuade me it would be a good idea to
do this! I might be persuaded that doing the opposite makes some sense
though (in other words importing an existing socket and using that).
The reason for this is that the underlying code uses an abstraction over
sockets and that is purely internal - there may not always be a socket
to expose and trying to do so would just make things hard for not a huge
gain in my opinion. However importing an existing socket would work
within the existing code structure to some extent.
BTW exposing the tcp-nodelay setting was intended to allow the user to
trade off latency against throughput, rather than to expose a socket
option specifically. I originally suggested a naming along those lines,
but tcp-nodelay stuck.
In the tests we've run, we found that tcp-nodelay is the better setting
in nearly every circumstance. So we've now made that the default setting
for the C++ code on trunk.
Andrew
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Single threaded Client
Posted by "Lance D." <la...@gmail.com>.
Thanks all for that advice. You all are exploring the same paths that I've
explored in some manner. I've got requirements that force my code to have
traceable paths, which makes multi-threading within the application
difficult. Right now, the option I like the best is probably the heartbeat
option solution.
On a side note, what are the chances that we could update the API to expose
the underlying socket and/or provide the ability to tweak the socket
options. The API already exposes the Nagle option, so I don't see that it
would be that blasphemous to add in other, similar options.
Thanks again!
-Lance
On Mon, May 6, 2013 at 10:16 AM, Bill Freeman <ke...@gmail.com> wrote:
> With respect to only the problem of attempting to make initial connection
> to a broker and handling the case in which the OS that broker is running on
> may be down:
>
> Since you are on a *nix box, you will have the epoll or select system call
> available to you. I believe that you can initiate your own (direct, not
> through the Qpid client library) socket connection to broker as an
> asynchronous operation (that is, tell the open not to wait for completion),
> and do this for all brokers in parallel. After a short timeout you can see
> who has responded with epoll or select. Machines that are up, with their
> networking enabled, and which are running a broker, will respond promptly,
> and be in the list of responders. You remove the others from your list of
> brokers to try. You now close ALL these connections (since I doubt that
> there is a way to pass the working ones off to the library). Then you can
> use the Qpid library to connect to the ones that are alive.
>
> You could argue that this is, in a sense, multi threaded in that multiple
> socket opens are flying at once, but it does not involve the use of kernel
> threads or a threading library in the client process (and the kernel itself
> is multi threaded whether you like it or not). You could, of course, make
> these test connections sequentially, specifying a short timeout (and not
> involve epoll or select at all), but the approach above gets you doing real
> work on live brokers that much sooner.
>
> Bill
>
>
> On Fri, May 3, 2013 at 5:06 PM, Andrew Stitcher <astitcher@redhat.com
> >wrote:
>
> > On Fri, 2013-05-03 at 08:14 -0400, Lance D. wrote:
> > > Hello,
> > >
> > > I've got a question about Socket timeouts. I've got a program that must
> > be
> > > single threaded. I know that's got its own share of problems, but the
> > > biggest one that I have right now is the case of trying to connect to a
> > > broker on a host that doesn't exist.
> > >
> > > I'll start with my configuration. I have a client running with the
> 0.14
> > > API on a Redhat Linux OS. My program uses a lookup service to find
> > brokers
> > > and exchanges that are providing data. The client connects to each
> > > exchange, pulls the data and disconnects. I don't need to do this
> > > extremely quickly, but it really shouldn't take more than 1-2 seconds
> per
> > > connection because they are all on a small network.
> > >
> > > My problems begin when the lookup service gives me a broker address
> for a
> > > host that is powered off. When I call open on the connection, the SYN
> > > packet is sent and I'm stuck waiting for the kernel socket timeout to
> > > expire before qpid throws that exception. That means that I'm stalled
> > for
> > > 20 seconds for each down host.
> > >
> > > So, my question to this wonderful group of users is this: what is the
> > best
> > > way (other than going multi-threaded), to reduce my wait from 20
> seconds
> > > down to 2 seconds?
> >
> > Assumptions:
> > * You are using the C++ messaging API (namespace qpid::messaging)
> >
> > I think you can solve this by setting the heartbeat timeout option on
> > the connection. If I remember correctly the client sets the heartbeat
> > timer before trying the connection so if it doesn't get a connection
> > within 2*heartbeat interval it will abort the connection.
> >
> > For example using the qpid-send application:
> >
> > qpid-send -a foo -b example.com
> >
> > Takes a long time to time out(your situation);
> >
> > qpid-send -a foo -b example.com -connection-options {heartbeat:1}
> >
> > Times out in 2 seconds (or so):
> > 2013-05-03 16:55:27 [Network] warning Connect failed: Connection
> > timedout
> > 2013-05-03 16:55:27 [Client] warning Connection closed
> > qpid-send: Failed to connect (reconnect disabled)
> >
> > programatically (untested snippets):
> > connection = Connection(url, "{heartbeat:1}");
> > connection.open();
> >
> > Or
> > connection.setOption("heartbeat", 1);
> > connection.open();
> >
> > Hope that helps.
> >
> > Andrew
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> > For additional commands, e-mail: users-help@qpid.apache.org
> >
> >
>
Re: Single threaded Client
Posted by Bill Freeman <ke...@gmail.com>.
With respect to only the problem of attempting to make initial connection
to a broker and handling the case in which the OS that broker is running on
may be down:
Since you are on a *nix box, you will have the epoll or select system call
available to you. I believe that you can initiate your own (direct, not
through the Qpid client library) socket connection to broker as an
asynchronous operation (that is, tell the open not to wait for completion),
and do this for all brokers in parallel. After a short timeout you can see
who has responded with epoll or select. Machines that are up, with their
networking enabled, and which are running a broker, will respond promptly,
and be in the list of responders. You remove the others from your list of
brokers to try. You now close ALL these connections (since I doubt that
there is a way to pass the working ones off to the library). Then you can
use the Qpid library to connect to the ones that are alive.
You could argue that this is, in a sense, multi threaded in that multiple
socket opens are flying at once, but it does not involve the use of kernel
threads or a threading library in the client process (and the kernel itself
is multi threaded whether you like it or not). You could, of course, make
these test connections sequentially, specifying a short timeout (and not
involve epoll or select at all), but the approach above gets you doing real
work on live brokers that much sooner.
Bill
On Fri, May 3, 2013 at 5:06 PM, Andrew Stitcher <as...@redhat.com>wrote:
> On Fri, 2013-05-03 at 08:14 -0400, Lance D. wrote:
> > Hello,
> >
> > I've got a question about Socket timeouts. I've got a program that must
> be
> > single threaded. I know that's got its own share of problems, but the
> > biggest one that I have right now is the case of trying to connect to a
> > broker on a host that doesn't exist.
> >
> > I'll start with my configuration. I have a client running with the 0.14
> > API on a Redhat Linux OS. My program uses a lookup service to find
> brokers
> > and exchanges that are providing data. The client connects to each
> > exchange, pulls the data and disconnects. I don't need to do this
> > extremely quickly, but it really shouldn't take more than 1-2 seconds per
> > connection because they are all on a small network.
> >
> > My problems begin when the lookup service gives me a broker address for a
> > host that is powered off. When I call open on the connection, the SYN
> > packet is sent and I'm stuck waiting for the kernel socket timeout to
> > expire before qpid throws that exception. That means that I'm stalled
> for
> > 20 seconds for each down host.
> >
> > So, my question to this wonderful group of users is this: what is the
> best
> > way (other than going multi-threaded), to reduce my wait from 20 seconds
> > down to 2 seconds?
>
> Assumptions:
> * You are using the C++ messaging API (namespace qpid::messaging)
>
> I think you can solve this by setting the heartbeat timeout option on
> the connection. If I remember correctly the client sets the heartbeat
> timer before trying the connection so if it doesn't get a connection
> within 2*heartbeat interval it will abort the connection.
>
> For example using the qpid-send application:
>
> qpid-send -a foo -b example.com
>
> Takes a long time to time out(your situation);
>
> qpid-send -a foo -b example.com -connection-options {heartbeat:1}
>
> Times out in 2 seconds (or so):
> 2013-05-03 16:55:27 [Network] warning Connect failed: Connection
> timedout
> 2013-05-03 16:55:27 [Client] warning Connection closed
> qpid-send: Failed to connect (reconnect disabled)
>
> programatically (untested snippets):
> connection = Connection(url, "{heartbeat:1}");
> connection.open();
>
> Or
> connection.setOption("heartbeat", 1);
> connection.open();
>
> Hope that helps.
>
> Andrew
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>
Re: Single threaded Client
Posted by Andrew Stitcher <as...@redhat.com>.
On Fri, 2013-05-03 at 08:14 -0400, Lance D. wrote:
> Hello,
>
> I've got a question about Socket timeouts. I've got a program that must be
> single threaded. I know that's got its own share of problems, but the
> biggest one that I have right now is the case of trying to connect to a
> broker on a host that doesn't exist.
>
> I'll start with my configuration. I have a client running with the 0.14
> API on a Redhat Linux OS. My program uses a lookup service to find brokers
> and exchanges that are providing data. The client connects to each
> exchange, pulls the data and disconnects. I don't need to do this
> extremely quickly, but it really shouldn't take more than 1-2 seconds per
> connection because they are all on a small network.
>
> My problems begin when the lookup service gives me a broker address for a
> host that is powered off. When I call open on the connection, the SYN
> packet is sent and I'm stuck waiting for the kernel socket timeout to
> expire before qpid throws that exception. That means that I'm stalled for
> 20 seconds for each down host.
>
> So, my question to this wonderful group of users is this: what is the best
> way (other than going multi-threaded), to reduce my wait from 20 seconds
> down to 2 seconds?
Assumptions:
* You are using the C++ messaging API (namespace qpid::messaging)
I think you can solve this by setting the heartbeat timeout option on
the connection. If I remember correctly the client sets the heartbeat
timer before trying the connection so if it doesn't get a connection
within 2*heartbeat interval it will abort the connection.
For example using the qpid-send application:
qpid-send -a foo -b example.com
Takes a long time to time out(your situation);
qpid-send -a foo -b example.com -connection-options {heartbeat:1}
Times out in 2 seconds (or so):
2013-05-03 16:55:27 [Network] warning Connect failed: Connection
timedout
2013-05-03 16:55:27 [Client] warning Connection closed
qpid-send: Failed to connect (reconnect disabled)
programatically (untested snippets):
connection = Connection(url, "{heartbeat:1}");
connection.open();
Or
connection.setOption("heartbeat", 1);
connection.open();
Hope that helps.
Andrew
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org