You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by spamtrap <no...@spamgourmet.com> on 2016/03/10 16:37:55 UTC
ActiveMQ-CPP Exception connecting to broker
[ActiveMQ-CPP v3.9.0]
We have a c++ program which connects to a broker, sends a message to a
topic and then closes the connections & exits. Every now and then it
fails to connect reporting a "Address already in use" exception
(based on ex.what()).
The code is like this:
----- cut ------
try {
pConnectionFactory =
cms::ConnectionFactory::createCMSConnectionFactory(
brokerURI);
pConnection = pConnectionFactory->createConnection();
} catch (cms::CMSException &ex)
{
fprintf(stderr, "CMSException: \n");
ex.printStackTrace();
...
}
----- cut ------
ex.printStackTrace prints nothing even though I am using the following
connection string (broker URI):
"tcp://localhost:61616?wireFormat.stackTraceEnabled=true"
How can I get more information about this problem?
Re: ActiveMQ-CPP Exception connecting to broker
Posted by "James A. Robinson" <ji...@highwire.org>.
Based on your numbers it's probably not ephemeral port exhaustion. The
default TIME_WAIT is 4 minutes on Solaris, and to use up 32,767 sockets
within 4 minutes, you'd need you to be sending an average of 136 messages
per second.
Jim
Re: ActiveMQ-CPP Exception connecting to broker
Posted by spamtrap <no...@spamgourmet.com>.
On Fri, 11 Mar 2016 14:53:09 +0000, "James A. Robinson"
<ji...@highwire.org> wrote:
>The TIME_WAIT state indicates both sides of the connection have shut down
>the socket cleanly and now the OS is waiting for some period of time before
>it will allow that socket to be reused. This is to prevent the
>possibility of a delayed packet on a previous session corrupting a
>subsequent session.
>
>When I asked earlier about how frequently "it performs this action" what I
>was intending to ask was "how frequently does your client send a message"?
I'd guess 2-4 per second (it's called by a script)
>If it sends many messages per second, you may be running out of sockets in
>the ephemeral port range. Are there tens of thousands of sockets showing
>up in netstat -nt (sockets of any state, not just TIME_WAIT)? I think
>Solaris typically allocates a range of 32k sockets for the client side of
>connections.
Only about 300-400 sockets in use.
>I think you can get the port range by using ndd to query the current values:
>
>/usr/sbin/ndd /dev/tcp tcp_smallest_anon_port tcp_largest_anon_port
32768
65535
>If you have at, or close to, <largest> - <smallest> sockets listed by
>netstat, you're probably running out of sockets in that range.
>
>If that's the case, I can think of a few possible solutions:
>
>1. increase the size of the range
>2. decrease the TIME_WAIT time
>3. redesign your client to use fewer sockets (e.g, batching messages)
>
>This might be helpful to read: http://www.sean.de/Solaris/soltune.html
Thanks - I'll read this.
>Anyway, this is just a theory based on the error message, it may very well
>be some other problem.
>
>On Fri, Mar 11, 2016 at 3:55 AM spamtrap <
>nospam.1.friedbadger@spamgourmet.com> wrote:
>
>> On Thu, 10 Mar 2016 16:13:32 +0000, "James A. Robinson"
>> <ji...@highwire.org> wrote:
>>
>> It happens infrequently and unpredictibly. For example sometimes it
>> fails after 25 times and sometimes it works for several 100 before
>> going wrong.
>>
>> Another thing is that program works OK on Linux. It's only on Solaris
>> that this problem arrises.
>>
>> netstat does show a lot of connections in TIME_WAIT state. I don't
>> know what control I have over the closedown -- I assume this is done
>> internally by ActiveMQ-CPP?
>>
Re: ActiveMQ-CPP Exception connecting to broker
Posted by "James A. Robinson" <ji...@highwire.org>.
The TIME_WAIT state indicates both sides of the connection have shut down
the socket cleanly and now the OS is waiting for some period of time before
it will allow that socket to be reused. This is to prevent the
possibility of a delayed packet on a previous session corrupting a
subsequent session.
When I asked earlier about how frequently "it performs this action" what I
was intending to ask was "how frequently does your client send a message"?
If it sends many messages per second, you may be running out of sockets in
the ephemeral port range. Are there tens of thousands of sockets showing
up in netstat -nt (sockets of any state, not just TIME_WAIT)? I think
Solaris typically allocates a range of 32k sockets for the client side of
connections.
I think you can get the port range by using ndd to query the current values:
/usr/sbin/ndd /dev/tcp tcp_smallest_anon_port tcp_largest_anon_port
If you have at, or close to, <largest> - <smallest> sockets listed by
netstat, you're probably running out of sockets in that range.
If that's the case, I can think of a few possible solutions:
1. increase the size of the range
2. decrease the TIME_WAIT time
3. redesign your client to use fewer sockets (e.g, batching messages)
This might be helpful to read: http://www.sean.de/Solaris/soltune.html
Anyway, this is just a theory based on the error message, it may very well
be some other problem.
On Fri, Mar 11, 2016 at 3:55 AM spamtrap <
nospam.1.friedbadger@spamgourmet.com> wrote:
> On Thu, 10 Mar 2016 16:13:32 +0000, "James A. Robinson"
> <ji...@highwire.org> wrote:
>
> It happens infrequently and unpredictibly. For example sometimes it
> fails after 25 times and sometimes it works for several 100 before
> going wrong.
>
> Another thing is that program works OK on Linux. It's only on Solaris
> that this problem arrises.
>
> netstat does show a lot of connections in TIME_WAIT state. I don't
> know what control I have over the closedown -- I assume this is done
> internally by ActiveMQ-CPP?
>
Re: ActiveMQ-CPP Exception connecting to broker
Posted by spamtrap <no...@spamgourmet.com>.
On Fri, 11 Mar 2016 07:19:27 -0700, Tim Bain
<tb...@alumni.duke.edu> wrote:
>I thought TIME_WAIT was the state after the application had closed the
>socket but before the OS had disposed it, so this may be a Solaris question.
My understanding (which may be wrong) is that the end which initiated
the closedown may end up in a TIME_WAIT state. In this case it should
be client rather than the server IMHO. If this was the situation then
I would not see this problem.
>On Mar 11, 2016 4:55 AM, "spamtrap" <no...@spamgourmet.com>
>wrote:
>
>> On Thu, 10 Mar 2016 16:13:32 +0000, "James A. Robinson"
>> <ji...@highwire.org> wrote:
>>
>> It happens infrequently and unpredictibly. For example sometimes it
>> fails after 25 times and sometimes it works for several 100 before
>> going wrong.
>>
>> Another thing is that program works OK on Linux. It's only on Solaris
>> that this problem arrises.
>>
>> netstat does show a lot of connections in TIME_WAIT state. I don't
>> know what control I have over the closedown -- I assume this is done
>> internally by ActiveMQ-CPP?
>>
>> >You don't say how frequently it performs this action? If it is frequent,
>> >and I were debugging it, the first thing I would check for would be that
>> >you haven't run out of socket filehandles. On a unix system running
>> >something like "netstat -nt" can tell you what state the socket
>> filehandles
>> >are in. If there are too many in the TIME_WAIT state, for example, it can
>> >exhaust your available filehandles and it won't be able to make a new
>> >socket connection until the OS allows them to expire. This exhaustion can
>> >easily happen if your program is only using a limited set of ports on the
>> >client side as well, and I could imagine that might be logged as an
>> >"address in use" error.
>> >
>> >Jim
>> >
>> >
>> >On Thu, Mar 10, 2016 at 8:07 AM Timothy Bish <ta...@gmail.com> wrote:
>> >
>> >> On 03/10/2016 10:37 AM, spamtrap wrote:
>> >> > [ActiveMQ-CPP v3.9.0]
>> >> >
>> >> > We have a c++ program which connects to a broker, sends a message to a
>> >> > topic and then closes the connections & exits. Every now and then it
>> >> > fails to connect reporting a "Address already in use" exception
>> >> > (based on ex.what()).
>> >> >
>> >> > The code is like this:
>> >> >
>> >> > ----- cut ------
>> >> > try {
>> >> > pConnectionFactory =
>> >> > cms::ConnectionFactory::createCMSConnectionFactory(
>> >> > brokerURI);
>> >> > pConnection = pConnectionFactory->createConnection();
>> >> > } catch (cms::CMSException &ex)
>> >> > {
>> >> > fprintf(stderr, "CMSException: \n");
>> >> > ex.printStackTrace();
>> >> > ...
>> >> > }
>> >> > ----- cut ------
>> >> >
>> >> > ex.printStackTrace prints nothing even though I am using the following
>> >> > connection string (broker URI):
>> >> > "tcp://localhost:61616?wireFormat.stackTraceEnabled=true"
>> >> >
>> >> > How can I get more information about this problem?
>> >> >
>> >> >
>> >> >
>> >> You'd need to do some debugging on the client end to see what is causing
>> >> this, not been reported before.
>> >>
>> >> --
>> >> Tim Bish
>> >> twitter: @tabish121
>> >> blog: http://timbish.blogspot.com/
>> >>
>> >>
>>
>>
>>
Re: ActiveMQ-CPP Exception connecting to broker
Posted by Tim Bain <tb...@alumni.duke.edu>.
I thought TIME_WAIT was the state after the application had closed the
socket but before the OS had disposed it, so this may be a Solaris question.
On Mar 11, 2016 4:55 AM, "spamtrap" <no...@spamgourmet.com>
wrote:
> On Thu, 10 Mar 2016 16:13:32 +0000, "James A. Robinson"
> <ji...@highwire.org> wrote:
>
> It happens infrequently and unpredictibly. For example sometimes it
> fails after 25 times and sometimes it works for several 100 before
> going wrong.
>
> Another thing is that program works OK on Linux. It's only on Solaris
> that this problem arrises.
>
> netstat does show a lot of connections in TIME_WAIT state. I don't
> know what control I have over the closedown -- I assume this is done
> internally by ActiveMQ-CPP?
>
> >You don't say how frequently it performs this action? If it is frequent,
> >and I were debugging it, the first thing I would check for would be that
> >you haven't run out of socket filehandles. On a unix system running
> >something like "netstat -nt" can tell you what state the socket
> filehandles
> >are in. If there are too many in the TIME_WAIT state, for example, it can
> >exhaust your available filehandles and it won't be able to make a new
> >socket connection until the OS allows them to expire. This exhaustion can
> >easily happen if your program is only using a limited set of ports on the
> >client side as well, and I could imagine that might be logged as an
> >"address in use" error.
> >
> >Jim
> >
> >
> >On Thu, Mar 10, 2016 at 8:07 AM Timothy Bish <ta...@gmail.com> wrote:
> >
> >> On 03/10/2016 10:37 AM, spamtrap wrote:
> >> > [ActiveMQ-CPP v3.9.0]
> >> >
> >> > We have a c++ program which connects to a broker, sends a message to a
> >> > topic and then closes the connections & exits. Every now and then it
> >> > fails to connect reporting a "Address already in use" exception
> >> > (based on ex.what()).
> >> >
> >> > The code is like this:
> >> >
> >> > ----- cut ------
> >> > try {
> >> > pConnectionFactory =
> >> > cms::ConnectionFactory::createCMSConnectionFactory(
> >> > brokerURI);
> >> > pConnection = pConnectionFactory->createConnection();
> >> > } catch (cms::CMSException &ex)
> >> > {
> >> > fprintf(stderr, "CMSException: \n");
> >> > ex.printStackTrace();
> >> > ...
> >> > }
> >> > ----- cut ------
> >> >
> >> > ex.printStackTrace prints nothing even though I am using the following
> >> > connection string (broker URI):
> >> > "tcp://localhost:61616?wireFormat.stackTraceEnabled=true"
> >> >
> >> > How can I get more information about this problem?
> >> >
> >> >
> >> >
> >> You'd need to do some debugging on the client end to see what is causing
> >> this, not been reported before.
> >>
> >> --
> >> Tim Bish
> >> twitter: @tabish121
> >> blog: http://timbish.blogspot.com/
> >>
> >>
>
>
>
Re: ActiveMQ-CPP Exception connecting to broker
Posted by spamtrap <no...@spamgourmet.com>.
On Thu, 10 Mar 2016 16:13:32 +0000, "James A. Robinson"
<ji...@highwire.org> wrote:
It happens infrequently and unpredictibly. For example sometimes it
fails after 25 times and sometimes it works for several 100 before
going wrong.
Another thing is that program works OK on Linux. It's only on Solaris
that this problem arrises.
netstat does show a lot of connections in TIME_WAIT state. I don't
know what control I have over the closedown -- I assume this is done
internally by ActiveMQ-CPP?
>You don't say how frequently it performs this action? If it is frequent,
>and I were debugging it, the first thing I would check for would be that
>you haven't run out of socket filehandles. On a unix system running
>something like "netstat -nt" can tell you what state the socket filehandles
>are in. If there are too many in the TIME_WAIT state, for example, it can
>exhaust your available filehandles and it won't be able to make a new
>socket connection until the OS allows them to expire. This exhaustion can
>easily happen if your program is only using a limited set of ports on the
>client side as well, and I could imagine that might be logged as an
>"address in use" error.
>
>Jim
>
>
>On Thu, Mar 10, 2016 at 8:07 AM Timothy Bish <ta...@gmail.com> wrote:
>
>> On 03/10/2016 10:37 AM, spamtrap wrote:
>> > [ActiveMQ-CPP v3.9.0]
>> >
>> > We have a c++ program which connects to a broker, sends a message to a
>> > topic and then closes the connections & exits. Every now and then it
>> > fails to connect reporting a "Address already in use" exception
>> > (based on ex.what()).
>> >
>> > The code is like this:
>> >
>> > ----- cut ------
>> > try {
>> > pConnectionFactory =
>> > cms::ConnectionFactory::createCMSConnectionFactory(
>> > brokerURI);
>> > pConnection = pConnectionFactory->createConnection();
>> > } catch (cms::CMSException &ex)
>> > {
>> > fprintf(stderr, "CMSException: \n");
>> > ex.printStackTrace();
>> > ...
>> > }
>> > ----- cut ------
>> >
>> > ex.printStackTrace prints nothing even though I am using the following
>> > connection string (broker URI):
>> > "tcp://localhost:61616?wireFormat.stackTraceEnabled=true"
>> >
>> > How can I get more information about this problem?
>> >
>> >
>> >
>> You'd need to do some debugging on the client end to see what is causing
>> this, not been reported before.
>>
>> --
>> Tim Bish
>> twitter: @tabish121
>> blog: http://timbish.blogspot.com/
>>
>>
Re: ActiveMQ-CPP Exception connecting to broker
Posted by "James A. Robinson" <ji...@highwire.org>.
You don't say how frequently it performs this action? If it is frequent,
and I were debugging it, the first thing I would check for would be that
you haven't run out of socket filehandles. On a unix system running
something like "netstat -nt" can tell you what state the socket filehandles
are in. If there are too many in the TIME_WAIT state, for example, it can
exhaust your available filehandles and it won't be able to make a new
socket connection until the OS allows them to expire. This exhaustion can
easily happen if your program is only using a limited set of ports on the
client side as well, and I could imagine that might be logged as an
"address in use" error.
Jim
On Thu, Mar 10, 2016 at 8:07 AM Timothy Bish <ta...@gmail.com> wrote:
> On 03/10/2016 10:37 AM, spamtrap wrote:
> > [ActiveMQ-CPP v3.9.0]
> >
> > We have a c++ program which connects to a broker, sends a message to a
> > topic and then closes the connections & exits. Every now and then it
> > fails to connect reporting a "Address already in use" exception
> > (based on ex.what()).
> >
> > The code is like this:
> >
> > ----- cut ------
> > try {
> > pConnectionFactory =
> > cms::ConnectionFactory::createCMSConnectionFactory(
> > brokerURI);
> > pConnection = pConnectionFactory->createConnection();
> > } catch (cms::CMSException &ex)
> > {
> > fprintf(stderr, "CMSException: \n");
> > ex.printStackTrace();
> > ...
> > }
> > ----- cut ------
> >
> > ex.printStackTrace prints nothing even though I am using the following
> > connection string (broker URI):
> > "tcp://localhost:61616?wireFormat.stackTraceEnabled=true"
> >
> > How can I get more information about this problem?
> >
> >
> >
> You'd need to do some debugging on the client end to see what is causing
> this, not been reported before.
>
> --
> Tim Bish
> twitter: @tabish121
> blog: http://timbish.blogspot.com/
>
>
Re: ActiveMQ-CPP Exception connecting to broker
Posted by Timothy Bish <ta...@gmail.com>.
On 03/10/2016 10:37 AM, spamtrap wrote:
> [ActiveMQ-CPP v3.9.0]
>
> We have a c++ program which connects to a broker, sends a message to a
> topic and then closes the connections & exits. Every now and then it
> fails to connect reporting a "Address already in use" exception
> (based on ex.what()).
>
> The code is like this:
>
> ----- cut ------
> try {
> pConnectionFactory =
> cms::ConnectionFactory::createCMSConnectionFactory(
> brokerURI);
> pConnection = pConnectionFactory->createConnection();
> } catch (cms::CMSException &ex)
> {
> fprintf(stderr, "CMSException: \n");
> ex.printStackTrace();
> ...
> }
> ----- cut ------
>
> ex.printStackTrace prints nothing even though I am using the following
> connection string (broker URI):
> "tcp://localhost:61616?wireFormat.stackTraceEnabled=true"
>
> How can I get more information about this problem?
>
>
>
You'd need to do some debugging on the client end to see what is causing
this, not been reported before.
--
Tim Bish
twitter: @tabish121
blog: http://timbish.blogspot.com/