"Too many open files" from C++ client

[Toralf Lund <to...@pgs.com>]

Just got the following from a client application using the C++ Messaging 
API:

2021-02-10 07:44:58 [Client] info Trying to connect to 
amqp:tcp:192.168.0.215:5672...
2021-02-10 07:44:58 [Network] debug TCPConnector created for 0-10
2021-02-10 07:44:58 [Client] debug Set TCP_NODELAY
2021-02-10 07:44:58 [System] info Connecting: 192.168.0.215:5672
2021-02-10 07:44:58 [System] debug Exception constructed: Too many open 
files 
/home/bmparr/rpmbuild/BUILD/srt-qpid-cpp-1.37/srt-qpid-cpp-1.37.0/src/qpid/sys/posix/BSDSocket.cpp:105)

This kept repeating itself until I restarted the process.

Any idea why this happened? What are these open "files"? Sockets 
representing connections to the server? Why would there be too many 
open? I suspect we're back to on old question here: How can I make sure 
all "stale" connections etc. go away? We also have situations were the 
client thought the connection was lost, but it is still there on the 
server-side according to "qpid-stat -c". They go away when restarting 
the client process, but I can't for the life of me figure out how to 
make the same thing happen through manipulation of the qpid objects...

The client is set up for auto-reconnect, but with a limited number of 
retries. There is also "manual" reconnect via a periodic check/update. 
It goes like this:

   if(!connection_.isOpen()) {
     /* Have partial exeption handling so execution is not interrupted if there
        is an error that isn't related to the connection as such. One case
        (the only I know of) is failure to reattach existing sessions - which
        Connection::open() tries to do after an automatic close on error. */
     try {
       connection_.open();
     } catch(const qpid::messaging::MessagingException &error) {
       if(!connection_.isOpen()) {
	throw;
       }
       Util::warning("\"%s\" when connecting to messaging server at %s",
		    error.what(), serverUrl_.c_str());
     }
   }
   if(session_.isValid() && session_.hasError()) {
     session_.close();
     session_=qpid::messaging::Session();
   }
   if(!session_.isValid()) {
     session_=connection_.createSession();
   }
}

Anything obviously wrong here? Is there something I should close or 
delete or reinitialise from scratch to make connections/open "files" go 
away?

Note that the same connection object is used throughout the lifetime of 
the application.

- Toralf