java.net.SocketTimeoutException downloading large files from tomcat

[Anthony Shaya <as...@workforcesoftware.com>]

Hello,

I'm using tomcat 8.5.15 and spring 4.3.3 framework for our server application (hosted on centos 6 machines). I have noticed we get the below exception (java.net.SocketTimeoutException) when people are downloading files from our server that are over 500 mb (many of the files are around a gb in size). Usually the behavior on the browser (firefox or chrome) is the download completes but the file is corrupt (zip in this case). The file size is off by 10mb each time. I've verified the actual files are valid on the servers file system. The only workaround I've found is to increase the connection timeout from 20seconds to 180 seconds (although I'm not sure why this would even matter). This seems to have fixed the issue, even though today we found another case after the timeout was increased where the issue resurfaced.


  *   This only seems to happen on our application when the downloads go over internet/network. Our local builds of the server application do not reproduce the problem.
  *   I've confirmed that firefox and chrome have the same issue and are not aborting the connection/download.
  *   I've attached the relevant stack trace, tomcat connector settings, relevant spring settings, and the code that does the sending of the file.


Any help or suggestions for troubleshooting would be much appreciated!


30-Jan-2018 15:16:23.130 SEVERE [http-nio-9000-exec-4] org.apache.catalina.core.StandardWrapperValve.invoke Servlet.service() for servlet [spring] in context with path [/] threw exception [Request processing failed; nested exception is java.lang.RuntimeException: Problem sending file distributions/ZIP_NAME.zip] with root cause
java.net.SocketTimeoutException
               at org.apache.tomcat.util.net.NioBlockingSelector.write(NioBlockingSelector.java:134)
               at org.apache.tomcat.util.net.NioSelectorPool.write(NioSelectorPool.java:157)
               at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doWrite(NioEndpoint.java:1259)
               at org.apache.tomcat.util.net.SocketWrapperBase.doWrite(SocketWrapperBase.java:670)
               at org.apache.tomcat.util.net.SocketWrapperBase.writeBlocking(SocketWrapperBase.java:450)
               at org.apache.tomcat.util.net.SocketWrapperBase.write(SocketWrapperBase.java:388)
               at org.apache.coyote.http11.Http11OutputBuffer$SocketOutputBuffer.doWrite(Http11OutputBuffer.java:644)
               at org.apache.coyote.http11.filters.IdentityOutputFilter.doWrite(IdentityOutputFilter.java:119)
               at org.apache.coyote.http11.Http11OutputBuffer.doWrite(Http11OutputBuffer.java:235)
               at org.apache.coyote.Response.doWrite(Response.java:568)
               at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:351)
               at org.apache.catalina.connector.OutputBuffer.flushByteBuffer(OutputBuffer.java:815)
               at org.apache.catalina.connector.OutputBuffer.append(OutputBuffer.java:720)
               at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:391)
               at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:369)
               at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:96)
               at org.springframework.security.web.context.SaveContextOnUpdateOrErrorResponseWrapper$SaveContextServletOutputStream.write(SaveContextOnUpdateOrErrorResponseWrapper.java:457)
               at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1793)
               at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1769)
               at org.apache.commons.io.IOUtils.copy(IOUtils.java:1744)


Below is my connector settings (fairly default)

    <Connector port="9000" secure="false" maxHttpHeaderSize="16384"
               minSpareThreads="25"
               maxThreads="75"
               acceptCount="250"
               enableLookups="false" redirectPort="8443"
               connectionTimeout="180000" disableUploadTimeout="true"
               asyncTimeout="300000"
               maxParameterCount="10000"
               compression="on"
               compressionMinSize="2048"
               compressableMimeType="text/html,text/xml,text/js,text/css"/>


Below is the spring bean we use for handling the multi part file upload
  <bean id="multipartResolver" class="org.springframework.web.multipart.commons.CommonsMultipartResolver"/>


  *   I'm aware of spring.http.multipart.max-file-size,pring.http.multipart.max-request-size and confirmed they are not set, resulting in the default of -1 being used for both.

Below is the actual code we use to send the file to the client

  public static void setDownloadContentHeaders(String fileName, boolean forceDownload, HttpServletResponse response, HttpServletRequest request) {
    response.setContentType(request.getServletContext().getMimeType(fileName));
    String contentDisposition = forceDownload ? "attachment" : "inline";
    response.setHeader("Content-Disposition", contentDisposition + "; filename=\"" + fileName + "\"");
  }

  public void sendFile(String path, boolean forceDownload, HttpServletRequest request, HttpServletResponse response) {
    try {
      // File to be downloaded
      File downloadFile = getFile(path); //this method just loads the file from disk
      if (!downloadFile.exists()) {
        response.sendError(HttpServletResponse.SC_NOT_FOUND);
        return;
      }

      try(InputStream fis = new FileInputStream(downloadFile)) {
        // Serve individual file
        setDownloadContentHeaders(downloadFile.getName(), forceDownload, response, request);
        if (downloadFile.length() < Integer.MAX_VALUE) {
          response.setContentLength((int)downloadFile.length());
        }
        IOUtils.copy(fis, response.getOutputStream());
      }
    } catch (IOException e) {
      throw new RuntimeException("Problem sending file " + path, e);
    }
  }




This message is intended exclusively for the individual or entity to which it is addressed. This communication may contain information that is proprietary, privileged, confidential or otherwise legally exempt from disclosure. If you are not the named addressee, or have been inadvertently and erroneously referenced in the address line, you are not authorized to read, print, retain, copy or disseminate this message or any part of it. If you have received this message in error, please notify the sender immediately by e-mail and delete all copies of the message. (ID m031214)