You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Martin Knoblauch <kn...@knobisoft.de> on 2020/08/19 09:58:16 UTC
How to upload Files larger than 2GB
Hi,
our customer has the following setup:
Apache/HTTPD(2.4.43)->mod_jk(1.2.48)->Tomcat(9.0.12).
The application hosted by Tomcat has a REST interface that allows file
upload using POST requests. The problem now is that we get a 500 response
when we try to upload files larger than 2 GB. But this happens only when
using the full path from Apache to Tomcat. When we try the upload directly
via the Tomcat HTTP port all is good
This leaves me with the question where the problem is. Is it the Tomcat AJP
part, is it mod_jk or is it HTTPD? From the logs I have the impression the
file never appears on the Tomcat side.
So I found the LimitRequestBody directive for HTTPD. But the description
leaves me wondering. It says "This directive specifies the number of bytes
from 0 (meaning unlimited) to 2147483647 (2GB) that are allowed in a
request body." Means "unlimited" really no limit, or is it 2GB. Anyway, I
set it to 0 with no success.
Then looking at the Tomcat configuration. The HTTP connector (working)
looks like this:
<Connector port="9695" protocol="HTTP/1.1"
connectionTimeout="20000"
maxPostSize="209715200"
redirectPort="8443" />
Which makes me wonder why it works. It should bail out at 200 MB. The AJP
connector looks like:
<Connector port="9697" protocol="AJP/1.3"
connectionTimeout="600000"
maxPostSize="20971520000"
maxThreads="300"
minSpareThreads="10"
redirectPort="8443" />
I upped the maxPostSize to 20GB, but it did not help.
Any advice is highly welcome. Just not
- Don't use HTTP for uploading large files. It is the mechanism the
application offers
- Don't allow upload of large files. Unfortunately it is a valid use-case.
Thanks in advance
Martin
--
------------------------------------------------------
Martin Knoblauch
email: k n o b i AT knobisoft DOT de
www: http://www.knobisoft.de
Re: How to upload Files larger than 2GB
Posted by Martin Knoblauch <kn...@googlemail.com.INVALID>.
Hi Mark,
thanks for the reply. The mentioning of "Chunked" helped me solve the
problem. Our perl based REST client clearly tried to send the file as one
buffer. Fortunately changing that was on line of code.
> >
> > Then looking at the Tomcat configuration. The HTTP connector (working)
> > looks like this:
> >
> > <Connector port="9695" protocol="HTTP/1.1"
> > connectionTimeout="20000"
> > maxPostSize="209715200"
> > redirectPort="8443" />
> >
> > Which makes me wonder why it works. It should bail out at 200 MB.
>
> That limit only applies to the automated processing of request bodies as
> per section 3.1.1 of the Servlet 4.0 specification.
>
> If the application (or a library it uses) reads the request body
> directly, there is no limit. The application is meant to provide
> whatever limits it considers appropriate.
>
>
Thanks for clarifying that.
> I wonder if httpd/mod_jk is trying cache the entire request body before
> forwarding. How is the request sent? With chunked-encoding or with a
> content-length? Does switching to the other one help?
>
>
Apparently with content-length. Changing to "chunked" solved it.
Thanks again
Martin
--
------------------------------------------------------
Martin Knoblauch
email: k n o b i AT knobisoft DOT de
www: http://www.knobisoft.de
[users@httpd] Re: How to upload Files larger than 2GB
Posted by Martin Knoblauch <kn...@googlemail.com.INVALID>.
Hi Mark,
thanks for the reply. The mentioning of "Chunked" helped me solve the
problem. Our perl based REST client clearly tried to send the file as one
buffer. Fortunately changing that was on line of code.
> >
> > Then looking at the Tomcat configuration. The HTTP connector (working)
> > looks like this:
> >
> > <Connector port="9695" protocol="HTTP/1.1"
> > connectionTimeout="20000"
> > maxPostSize="209715200"
> > redirectPort="8443" />
> >
> > Which makes me wonder why it works. It should bail out at 200 MB.
>
> That limit only applies to the automated processing of request bodies as
> per section 3.1.1 of the Servlet 4.0 specification.
>
> If the application (or a library it uses) reads the request body
> directly, there is no limit. The application is meant to provide
> whatever limits it considers appropriate.
>
>
Thanks for clarifying that.
> I wonder if httpd/mod_jk is trying cache the entire request body before
> forwarding. How is the request sent? With chunked-encoding or with a
> content-length? Does switching to the other one help?
>
>
Apparently with content-length. Changing to "chunked" solved it.
Thanks again
Martin
--
------------------------------------------------------
Martin Knoblauch
email: k n o b i AT knobisoft DOT de
www: http://www.knobisoft.de
Re: How to upload Files larger than 2GB
Posted by Mark Thomas <ma...@apache.org>.
On 19/08/2020 10:58, Martin Knoblauch wrote:
> Hi,
>
> our customer has the following setup:
>
> Apache/HTTPD(2.4.43)->mod_jk(1.2.48)->Tomcat(9.0.12).
>
> The application hosted by Tomcat has a REST interface that allows file
> upload using POST requests. The problem now is that we get a 500 response
> when we try to upload files larger than 2 GB. But this happens only when
> using the full path from Apache to Tomcat. When we try the upload directly
> via the Tomcat HTTP port all is good
>
> This leaves me with the question where the problem is. Is it the Tomcat AJP
> part, is it mod_jk or is it HTTPD? From the logs I have the impression the
> file never appears on the Tomcat side.
What logs?
If the request reached Tomcat it should appear in the access log and a
Tomcat generated 500 response should result in a stack trace in the logs.
The mod_jk log will also confirm what is and is not sent to Tomcat.
> So I found the LimitRequestBody directive for HTTPD. But the description
> leaves me wondering. It says "This directive specifies the number of bytes
> from 0 (meaning unlimited) to 2147483647 (2GB) that are allowed in a
> request body." Means "unlimited" really no limit, or is it 2GB. Anyway, I
> set it to 0 with no success.
>
> Then looking at the Tomcat configuration. The HTTP connector (working)
> looks like this:
>
> <Connector port="9695" protocol="HTTP/1.1"
> connectionTimeout="20000"
> maxPostSize="209715200"
> redirectPort="8443" />
>
> Which makes me wonder why it works. It should bail out at 200 MB.
That limit only applies to the automated processing of request bodies as
per section 3.1.1 of the Servlet 4.0 specification.
If the application (or a library it uses) reads the request body
directly, there is no limit. The application is meant to provide
whatever limits it considers appropriate.
> The AJP
> connector looks like:
>
> <Connector port="9697" protocol="AJP/1.3"
> connectionTimeout="600000"
> maxPostSize="20971520000"
> maxThreads="300"
> minSpareThreads="10"
> redirectPort="8443" />
>
> I upped the maxPostSize to 20GB, but it did not help.
Same as for HTTP above.
> Any advice is highly welcome. Just not
>
> - Don't use HTTP for uploading large files. It is the mechanism the
> application offers
> - Don't allow upload of large files. Unfortunately it is a valid use-case.
Confirm with the mod_jk log whether the request is sent to Tomcat.
I wonder if httpd/mod_jk is trying cache the entire request body before
forwarding. How is the request sent? With chunked-encoding or with a
content-length? Does switching to the other one help?
You might need to take a look at the mod_jk (or httpd) source to see how
request bodies are handled.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org