You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tomcat.apache.org by bu...@apache.org on 2007/05/30 20:27:10 UTC
DO NOT REPLY [Bug 42546] New: - Cannot send a Content-Length header larger than 2GB when connecting to Tomcat with Mod_JK
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG�
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://issues.apache.org/bugzilla/show_bug.cgi?id=42546>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND�
INSERTED IN THE BUG DATABASE.
http://issues.apache.org/bugzilla/show_bug.cgi?id=42546
Summary: Cannot send a Content-Length header larger than 2GB when
connecting to Tomcat with Mod_JK
Product: Tomcat 5
Version: 5.5.23
Platform: PC
OS/Version: Linux
Status: NEW
Severity: major
Priority: P2
Component: Connector:AJP
AssignedTo: tomcat-dev@jakarta.apache.org
ReportedBy: pmcmahon-apachebugzilla@perpetual.com
Summary
-------
I am having some trouble using Mod_Jk to serve dynamically generated content
that is larger than 2GB. The file is sent successfully, but the Content-Length
header is not set properly, forcing chunked encoding. The client downloading
this file is then not given an indication of how long the transfer will take to
complete. If I use Mod_Proxy to serve the file, or connect directly to Tomcat's
HTTP1.1 connector, the Content-Length is set properly and the client can follow
the download progress to completion.
In my own testing, it seems the Content-Length header is wrapping beyond 2^31,
and when Tomcat's AJP connector constructs the AJP packets, it sees a negative
Content-Length header and drops this, setting the Content-Length to zero and
forcing chunked encoding. I don't have enough Java knowledge to understand the
code in question, so hopefully some test cases and a sample webapp will provide
enough of a starting point to resolve the issue.
Software
--------
This test was conducted on 64bit AMD boxes on both a regular RedHat Enterprise
Linux 4 update 4 machine and a a Xen instance of the same software.
~ # uname -a
Linux sandbox 2.6.12.6-xenU #1 SMP Thu Jul 6 23:24:02 GMT 2006 x86_64 x86_64
x86_64 GNU/Linux
~ # uname -a
Linux testbox 2.6.9-42.0.3.ELsmp #1 SMP Mon Sep 25 17:24:31 EDT 2006 x86_64
x86_64 x86_64 GNU/Linux
Tomcat is the Core binary distribution of 5.5.23 running on the 64bit version of
Java 1.5.0_10 (also tested with 32bit).
~ # java -version
java version "1.5.0_10"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_10-b03)
Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_10-b03, mixed mode)
To test this through Apache, I've used RedHat's official version of Apache
2.0.52-28.ent. The MPM has been overridden in /etc/sysconfig/httpd to use the
worker MPM (although this issue occurs with the default prefork MPM).
~ # httpd -V
Server version: Apache/2.0.52
Server built: Aug 2 2006 05:21:18
Server's Module Magic Number: 20020903:9
Architecture: 64-bit
Server compiled with....
-D APACHE_MPM_DIR="server/mpm/prefork"
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_SYSVSEM_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D HTTPD_ROOT="/etc/httpd"
-D SUEXEC_BIN="/usr/sbin/suexec"
-D DEFAULT_PIDLOG="logs/httpd.pid"
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
-D DEFAULT_LOCKFILE="logs/accept.lock"
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="conf/mime.types"
-D SERVER_CONFIG_FILE="conf/httpd.conf"
RedHat's package includes Mod_Proxy. I built Mod_JK from source using the 1.2.23
release. Mod_JK and Tomcat are not using JNI. Apache loads Mod_JK's DSO through
the /etc/httpd/conf.d/mod_jk.conf file, which itself is sourced by Apache's
httpd.conf.
# Load the module
LoadModule jk_module modules/mod_jk.so
# Location of the workers.properties file
JkWorkersFile conf/jk-workers.properties
# Where to put jk shared memory
JkShmFile logs/jk-runtime-status
# Log file location
JkLogFile logs/mod_jk.log
# Log level: debug, error, info
JkLogLevel debug
# Log file timestamp format for JkLogFile
#JkLogStampFormat "[%a %b %d %H:%M:%S %Y] "
# Forwarding options
JkOptions +ForwardKeySize +ForwardURICompat -ForwardDirectories
# Environment variables passed into Tomcat from Apache
#JkEnvVar SSL_CLIENT_V_START
# Request log format
JkRequestLogFormat "%w %V %T"
The jk-workers.properties file contains an example configuration for a load
balanced pair of Tomcat servers, but in this case only the first Tomcat server
is added to the worker list.
# This worker will be used by Apache
worker.list=loadbalancer
# Worker type is loadbalanced
worker.loadbalancer.type=lb
# Comma separated list of loadbalanced workers
# By default, we only use the first worker
worker.loadbalancer.balanced_workers=tomcat1
# First Tomcat server worker
worker.tomcat1.port=8009
worker.tomcat1.host=localhost.localdomain
worker.tomcat1.type=ajp13
worker.tomcat1.lbfactor=1
worker.tomcat1.cachesize=10
worker.tomcat1.route=localhost
# Second Tomcat server worker
# To enable, add "tomcat2" to the balanced_workers directive above
worker.tomcat2.port=8009
worker.tomcat2.host=otherserver.localdomain
worker.tomcat2.type=ajp13
worker.tomcat2.lbfactor=1
worker.tomcat2.cachesize=10
worker.tomcat2.route=otherserver
# Path separator
ps=/
Apache is configured to listen on port 81 and port 82 for these specific tests.
Port 81 will be served via Mod_Proxy to Tomcat's HTTP1.1 connector, and port 82
will be served via Mod_JK to Tomcat's AJP connector. These configuration
directives were added to /etc/httpd/conf/httpd.conf:
# Test cases
# Port 81 will be served via mod_proxy
Listen 81
NameVirtualHost *:81
<VirtualHost *:81>
ProxyRequests off
ProxyPreserveHost On
ProxyPass / http://localhost:8080/
ProxyPassReverse / http://localhost:8080/
</Virtualhost>
# Port 82 will be served via mod_jk
Listen 82
NameVirtualHost *:82
<VirtualHost *:82>
JkMount / loadbalancer
JkMount /* loadbalancer
</VirtualHost>
# Note that we'll also test with 8080
# Which will be served directly by Tomcat
Test Cases
==========
Test cases are done using a small webapp that generates a 3GB file filled with
"A" characters. I will attach this webapp with source code. I believe a similar
test can be run with Tomcat serving a static 3GB file from it's document root. I
didn't test this as I will be serving files dynamically generated by a webapp
(in a similar way that this webapp generates the 3GB file). Download (and
header) tests were performed via curl on a remote machine (Ubuntu Feisty Fawn
x86_64). Packet captures were done using Ethereal/Wireshark.
~ $ curl -V
curl 7.15.5 (x86_64-pc-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8c zlib/1.2.3
libidn/0.6.5
Protocols: tftp ftp telnet dict ldap http file https ftps
Features: GSS-Negotiate IDN IPv6 Largefile NTLM SSL libz
First, I verified the Content-Length is being properly when I connected directly
to Tomcat's HTTP1.1 connector in port 8080. Note that the content itself is
truncated in these examples.
~ $ curl -D headers -o bigfile http://sandbox:8080/WebLoad/webload
This shows the proper header length, and I'm given a time to completion:
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 3192M 0 10.5M 0 0 10.8M 0 0:04:53 --:--:-- 0:04:53 11.0M
Ethereal shows the following connection (by selecting Follow TCP Stream):
GET /WebLoad/webload HTTP/1.1
User-Agent: curl/7.15.5 (x86_64-pc-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8c
zlib/1.2.3 libidn/0.6.5
Host: sandbox:8080
Accept: */*
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Content-Type: application/octet-stream
Content-Length: 3347054592
Date: Tue, 29 May 2007 18:56:29 GMT
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
Next, I tested by using Mod_Proxy on port 81.
~ $ curl -D headers -o bigfile http://sandbox:81/WebLoad/webload
This returns the Content-Length and the progress of the download is properly
reported. Ethereal shows:
GET /WebLoad/webload HTTP/1.1
User-Agent: curl/7.15.5 (x86_64-pc-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8c
zlib/1.2.3 libidn/0.6.5
Host: sandbox:81
Accept: */*
HTTP/1.1 200 OK
Date: Wed, 30 May 2007 16:43:20 GMT
Server: Apache-Coyote/1.1
Content-Type: application/octet-stream
Content-Length: 3347054592
Connection: close
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
Finally, I test the download through the AJP connector/Mod_JK.
~ $ curl -D headers -o bigfile http://sandbox:82/WebLoad/webload
GET /WebLoad/webload HTTP/1.1
User-Agent: curl/7.15.5 (x86_64-pc-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8c
zlib/1.2.3 libidn/0.6.5
Host: sandbox:82
Accept: */*
HTTP/1.1 200 OK
Date: Tue, 29 May 2007 19:06:35 GMT
Server: Apache/2.0.52 (Red Hat)
Connection: close
Transfer-Encoding: chunked
Content-Type: application/octet-stream
1ff8
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
Note that there is no Content-Length header, and the connection is forced to use
chunked encoding (with a chunk size prefixing the data). Apache is not getting a
Content-Length header, as show in the entries from the Mod_JK debug log.
[Wed May 30 17:03:05 2007] [17422:6496] [debug]
ajp_get_endpoint::jk_ajp_common.c (2343): acquired connection pool slot=0
[Wed May 30 17:03:05 2007] [17422:6496] [debug]
ajp_marshal_into_msgb::jk_ajp_common.c (548): ajp marshaling done
[Wed May 30 17:03:05 2007] [17422:6496] [debug] ajp_service::jk_ajp_common.c
(1796): processing tomcat1 with 2 retries
[Wed May 30 17:03:05 2007] [17422:6496] [debug] jk_open_socket::jk_connect.c
(335): socket TCP_NODELAY set to On
[Wed May 30 17:03:05 2007] [17422:6496] [debug] jk_open_socket::jk_connect.c
(433): trying to connect socket 14 to 127.0.0.1:8009
[Wed May 30 17:03:05 2007] [17422:6496] [debug] jk_open_socket::jk_connect.c
(459): socket 14 connected to 127.0.0.1:8009
[Wed May 30 17:03:05 2007] [17422:6496] [debug]
ajp_connect_to_endpoint::jk_ajp_common.c (850): Connected socket 14 to
(127.0.0.1:8009)
[Wed May 30 17:03:05 2007] [17422:6496] [debug]
ajp_connection_tcp_send_message::jk_ajp_common.c (896): sending to ajp13 pos=4
len=201 max=8192
[Wed May 30 17:03:05 2007] [17422:6496] [debug]
ajp_connection_tcp_send_message::jk_ajp_common.c (896): 0000 12 34 00 C5 02
02 00 08 48 54 54 50 2F 31 2E 31 - .4......HTTP/1.1
[Wed May 30 17:03:05 2007] [17422:6496] [debug]
ajp_connection_tcp_send_message::jk_ajp_common.c (896): 0010 00 00 10 2F 57
65 62 4C 6F 61 64 2F 77 65 62 6C - .../WebLoad/webl
[Wed May 30 17:03:05 2007] [17422:6496] [debug]
ajp_connection_tcp_send_message::jk_ajp_common.c (896): 0020 6F 61 64 00 00
0A 31 30 2E 38 2E 36 2E 31 33 36 - oad...10.8.6.136
[Wed May 30 17:03:05 2007] [17422:6496] [debug]
ajp_connection_tcp_send_message::jk_ajp_common.c (896): 0030 00 FF FF 00 07
73 61 6E 64 62 6F 78 00 00 52 00 - .....sandbox..R.
[Wed May 30 17:03:05 2007] [17422:6496] [debug]
ajp_connection_tcp_send_message::jk_ajp_common.c (896): 0040 00 04 A0 0E 00
57 63 75 72 6C 2F 37 2E 31 35 2E - .....Wcurl/7.15.
[Wed May 30 17:03:05 2007] [17422:6496] [debug]
ajp_connection_tcp_send_message::jk_ajp_common.c (896): 0050 35 20 28 78 38
36 5F 36 34 2D 70 63 2D 6C 69 6E - 5.(x86_64-pc-lin
[Wed May 30 17:03:05 2007] [17422:6496] [debug]
ajp_connection_tcp_send_message::jk_ajp_common.c (896): 0060 75 78 2D 67 6E
75 29 20 6C 69 62 63 75 72 6C 2F - ux-gnu).libcurl/
[Wed May 30 17:03:05 2007] [17422:6496] [debug]
ajp_connection_tcp_send_message::jk_ajp_common.c (896): 0070 37 2E 31 35 2E
35 20 4F 70 65 6E 53 53 4C 2F 30 - 7.15.5.OpenSSL/0
[Wed May 30 17:03:05 2007] [17422:6496] [debug]
ajp_connection_tcp_send_message::jk_ajp_common.c (896): 0080 2E 39 2E 38 63
20 7A 6C 69 62 2F 31 2E 32 2E 33 - .9.8c.zlib/1.2.3
[Wed May 30 17:03:05 2007] [17422:6496] [debug]
ajp_connection_tcp_send_message::jk_ajp_common.c (896): 0090 20 6C 69 62 69
64 6E 2F 30 2E 36 2E 35 00 A0 0B - .libidn/0.6.5...
[Wed May 30 17:03:05 2007] [17422:6496] [debug]
ajp_connection_tcp_send_message::jk_ajp_common.c (896): 00a0 00 0A 73 61 6E
64 62 6F 78 3A 38 32 00 A0 01 00 - ..sandbox:82....
[Wed May 30 17:03:05 2007] [17422:6496] [debug]
ajp_connection_tcp_send_message::jk_ajp_common.c (896): 00b0 03 2A 2F 2A 00
A0 08 00 01 30 00 06 00 09 6C 6F - .*/*.....0....lo
[Wed May 30 17:03:05 2007] [17422:6496] [debug]
ajp_connection_tcp_send_message::jk_ajp_common.c (896): 00c0 63 61 6C 68 6F
73 74 00 FF 00 00 00 00 00 00 00 - calhost.........
[Wed May 30 17:03:05 2007] [17422:6496] [debug]
ajp_send_request::jk_ajp_common.c (1287): (tomcat1) request body to send 0 -
request body to resend 0
[Wed May 30 17:03:05 2007] [17422:6496] [debug]
ajp_connection_tcp_get_message::jk_ajp_common.c (1043): received from ajp13
pos=0 len=52 max=8192
[Wed May 30 17:03:05 2007] [17422:6496] [debug]
ajp_connection_tcp_get_message::jk_ajp_common.c (1043): 0000 04 00 C8 00 02
4F 4B 00 00 01 00 0C 43 6F 6E 74 - .....OK.....Cont
[Wed May 30 17:03:05 2007] [17422:6496] [debug]
ajp_connection_tcp_get_message::jk_ajp_common.c (1043): 0010 65 6E 74 2D 54
79 70 65 00 00 18 61 70 70 6C 69 - ent-Type...appli
[Wed May 30 17:03:05 2007] [17422:6496] [debug]
ajp_connection_tcp_get_message::jk_ajp_common.c (1043): 0020 63 61 74 69 6F
6E 2F 6F 63 74 65 74 2D 73 74 72 - cation/octet-str
[Wed May 30 17:03:05 2007] [17422:6496] [debug]
ajp_connection_tcp_get_message::jk_ajp_common.c (1043): 0030 65 61 6D 00 00
00 00 00 00 00 00 00 00 00 00 00 - eam.............
[Wed May 30 17:03:05 2007] [17422:6496] [debug]
ajp_unmarshal_response::jk_ajp_common.c (603): status = 200
[Wed May 30 17:03:05 2007] [17422:6496] [debug]
ajp_unmarshal_response::jk_ajp_common.c (610): Number of headers is = 1
[Wed May 30 17:03:05 2007] [17422:6496] [debug]
ajp_unmarshal_response::jk_ajp_common.c (666): Header[0] [Content-Type] =
[application/octet-stream]
[Wed May 30 17:03:05 2007] [17422:6496] [debug]
ajp_connection_tcp_get_message::jk_ajp_common.c (1043): received from ajp13
pos=0 len=8188 max=8192
[Wed May 30 17:03:05 2007] [17422:6496] [debug]
ajp_connection_tcp_get_message::jk_ajp_common.c (1043): 0000 03 1F F8 41 41
41 41 41 41 41 41 41 41 41 41 41 - ...AAAAAAAAAAAAA
[Wed May 30 17:03:05 2007] [17422:6496] [debug]
ajp_connection_tcp_get_message::jk_ajp_common.c (1043): 0010 41 41 41 41 41
41 41 41 41 41 41 41 41 41 41 41 - AAAAAAAAAAAAAAAA
[Wed May 30 17:03:05 2007] [17422:6496] [debug]
ajp_connection_tcp_get_message::jk_ajp_common.c (1043): 0020 41 41 41 41 41
41 41 41 41 41 41 41 41 41 41 41 - AAAAAAAAAAAAAAAA
[Wed May 30 17:03:05 2007] [17422:6496] [debug]
ajp_connection_tcp_get_message::jk_ajp_common.c (1043): 0030 41 41 41 41 41
41 41 41 41 41 41 41 41 41 41 41 - AAAAAAAAAAAAAAAA
If I run Ethereal on the Tomcat server itself and capture the AJP packets, I see
the same information. Here is the AJP packet from Apache:
4......HTTP/1.1.../WebLoad/webload..
10.8.6.136.....sandbox..R......Wcurl/7.15.5 (x86_64-pc-linux-gnu) libcurl/7.15.5
OpenSSL/0.9.8c zlib/1.2.3 libidn/0.6.5....
sandbox:82.....*/*.....0....localhost..
And from Tomcat:
AB.4.....OK.....Content-Type...application/octet-stream.AB.....AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
Again, no Content-Length is being set.
Notes
=====
Mod_Proxy will not work for the way I am using Tomcat to serve data, as
Mod_Proxy strips the end user's client IP when connecting to Tomcat. In this
example, all end users downloading these files would appear to be connecting
from 127.0.0.1. Mod_JK solves this issue by including the end user's IP in the
headers, so webapps can access this and use it for managing security access, etc.
I'm filing this bug under Platform:PC and OS:Linux because that's what I've
tested, but I believe this bug may affect other platforms as well.
--
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
DO NOT REPLY [Bug 42546] - Cannot send a Content-Length header larger than 2GB when connecting to Tomcat with Mod_JK
Posted by bu...@apache.org.
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG�
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://issues.apache.org/bugzilla/show_bug.cgi?id=42546>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND�
INSERTED IN THE BUG DATABASE.
http://issues.apache.org/bugzilla/show_bug.cgi?id=42546
rainer.jung@kippdata.de changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |RESOLVED
Resolution| |FIXED
------- Additional Comments From rainer.jung@kippdata.de 2008-01-01 18:04 -------
Please check, if this is still a problem. A couple of problems with large file
transfers have been fixed during 2007 in Tomcat and in mod_jk. For details see
BZ 42608.
--
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
DO NOT REPLY [Bug 42546] - Cannot send a Content-Length header larger than 2GB when connecting to Tomcat with Mod_JK
Posted by bu...@apache.org.
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG�
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://issues.apache.org/bugzilla/show_bug.cgi?id=42546>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND�
INSERTED IN THE BUG DATABASE.
http://issues.apache.org/bugzilla/show_bug.cgi?id=42546
------- Additional Comments From pmcmahon-apachebugzilla@perpetual.com 2007-05-30 11:30 -------
Created an attachment (id=20289)
--> (http://issues.apache.org/bugzilla/attachment.cgi?id=20289&action=view)
Including the WebLoad.war file, which dynamically serves a 3GB file full of "A"
characters.
This webapp requires log4j in Tomcat's common/lib directory. Download a 3GB
file by going hitting this URL on the Tomcat server:
/WebLoad/webload
--
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
DO NOT REPLY [Bug 42546] - Cannot send a Content-Length header larger than 2GB when connecting to Tomcat with Mod_JK
Posted by bu...@apache.org.
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG�
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://issues.apache.org/bugzilla/show_bug.cgi?id=42546>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND�
INSERTED IN THE BUG DATABASE.
http://issues.apache.org/bugzilla/show_bug.cgi?id=42546
------- Additional Comments From pmcmahon-apachebugzilla@perpetual.com 2007-05-30 11:31 -------
Created an attachment (id=20290)
--> (http://issues.apache.org/bugzilla/attachment.cgi?id=20290&action=view)
Including WebLoad.war source code.
I don't have build steps for this .war file as the engineer who wrote it did
not provide them.
--
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org