[NET] java.net.SocketException: Too many open files

[David Griffiths <dg...@boats.com>]

I've searched the web, and through the archives, looking for similar problems. The most common cause seems to be too few file 
descriptors.

We seem to have lots of file descriptors:

cat /proc/sys/fs/file-max
406054

And not many are being used (this is with the code running):

cat /proc/sys/fs/file-nr
2730    0       406054

This happens in a long-running program (it processes images, and keeps going till there are no more - there are usually a few
thousand in the queue waiting to be downloaded).

A group of images is retrieved via http (wget), resized (via image-magik-convert), and then ftp'd up to other servers.

The exception always occurs in the ftp "storeFile" portion of the process.

The code is,

try
{
	File remoteFile = new File(strRemoteDirectory + f.getName());
	boolean bOneSucceeded = false;
	for (int iServerCount = 0; iServerCount < ftpConnections.length; iServerCount++)
         {
                 FTPClient ftpClient = ftpConnections[iServerCount];

                 if ( !ftpClient.changeWorkingDirectory(strRemoteDirectory) )
                 {
                 	m_sLogger.info("\t" + new java.util.Date() + " ListingExchangeImageProcessor: run: could not change directories
(on ftp server) to " + strRemoteDirectory);
                         continue;
		}

                 fInputStream = new FileInputStream(f);
		ftpClient.setFileType(FTP.BINARY_FILE_TYPE);

		if ( ftpClient.storeFile(f.getName(), fInputStream) )
                 {
                 	bOneSucceeded = true;
		}

	}

         if ( !bOneSucceeded )
         {
         	m_sLogger.info("\t" + new java.util.Date() + " ListingExchangeImageProcessor: run: unable to upload to a single ftp
server...");

		return;
	}
}
catch (Exception e)
{
	m_sLogger.info("\t" + new java.util.Date() + " ListingExchangeImageProcessor: run: exception uploading file: " + e.toString());
         e.printStackTrace();
         return;
}
finally
{
	try
         {
         	fInputStream.close();
	}
         catch (Exception e) {}
}

The stack trace for the exception is,

java.net.SocketException: Too many open files
         at java.net.PlainSocketImpl.socketAccept(Native Method)
         at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
         at java.net.ServerSocket.implAccept(ServerSocket.java:450)
         at java.net.ServerSocket.accept(ServerSocket.java:421)
         at org.apache.commons.net.ftp.FTPClient._openDataConnection_(FTPClient.java:502)
         at org.apache.commons.net.ftp.FTPClient.__storeFile(FTPClient.java:388)
         at org.apache.commons.net.ftp.FTPClient.storeFile(FTPClient.java:1388)
         at listingexchange.ListingExchangeImageProcessor.run(ListingExchangeImageProcessor.java:1143)
         at listingexchange.ListingExchangeImageProcessor.main(ListingExchangeImageProcessor.java:204)


Any insight would be helpful.

David

---------------------------------------------------------------------
To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-user-help@jakarta.apache.org

Re: [NET] java.net.SocketException: Too many open files

[David Griffiths <dg...@boats.com>]

I pass a comma-delimited list of servers (image servers) into the program; it's parsed, and then the # of image servers is used to 
create an array of ftp connections.


I've also discovered that there are limits to the number of sockets that the user this runs under can access (1024). I just bumped 
it up to 4096.

Wish I'd discovered that 15 minutes earlier. Please disregard my post.

David


Laurent Perez wrote:
>>                  FTPClient ftpClient = ftpConnections[iServerCount];
> 
> Out of curiosity, how is ftpConnections array populated ?
> 
> laurent
> 
> --
> <a href="http://in-pocket.blogspot.com">http://in-pocket.blogspot.com
> - Mobile world, technology and more</a>
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: commons-user-help@jakarta.apache.org
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-user-help@jakarta.apache.org

Re: [NET] java.net.SocketException: Too many open files

[Laurent Perez <ha...@gmail.com>]

>                  FTPClient ftpClient = ftpConnections[iServerCount];

Out of curiosity, how is ftpConnections array populated ?

laurent

--
<a href="http://in-pocket.blogspot.com">http://in-pocket.blogspot.com
- Mobile world, technology and more</a>

---------------------------------------------------------------------
To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-user-help@jakarta.apache.org