You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Tero Karttunen <ka...@gmail.com> on 2010/01/11 14:41:14 UTC
mod_jk: plus-character causes %-encoding problems
SHORT SUMMARY:
===============
When Apache HTTPD is used for both URL manipulation (request
redirecting) and load-balancing with mod_jk, the plus-characters seem
to behave strangely during character re-encoding. Is this a bug or a
feature, and how should the system be set up to ensure correct
operation?
MY SYSTEM SET-UP:
=================
My application is running on two separate Tomcat 6.0.20 instances.
Apache HTTPD 2.2.14 (with mod_jk 1.2.28) acts as facade and load
balancer for my application. Apache HTTPD performs two functions: it
performs URL redirection from short URLs /sites/* to longer actual
URLs /contextroot/subcontext/sites/, and it forwards the requests to
the actual URLs to the loadbalancer, which in turn utilizes the Tomcat
instances via AJP connectors. The redirect requests are being served
by mod_alias, not mod_rewrite.
Here are the most relevant lines in httpd.conf:
LogFormat "%a %t \"%r\" %X%>s
%{JK_LB_LAST_NAME}n(%{JK_LB_FIRST_NAME}n) %b %D" ts
CustomLog "|bin/rotatelogs logs/access-%Y%m%d-%H%M%S.log 5M" ts
JkMount /contextroot/* loadbalancer
AllowEncodedSlashes On
RedirectMatch ^/sites(.*)$ /contextroot/subcontext/sites$1
Notice that I have not set any explicit forwarding options, which in
this mod_jk version implies +ForwardURIProxy.
To begin with: Calling Tomcat HTTP connectors directly with
http://localhost:8082/contextroot/subcontext/sites/one%2Bone%3Cthree
work correctly and application decodes the URL to
"sites/one+one<three" (with the correct preceding context path of
course). The problems begin while I try to access the application
through Apache HTTPD.
OBSERVED BEHAVIOR WITH +ForwardURIProxy: (implicit)
=====================================
I try to access URL http://localhost/sites/one%2Bone%3Cthree with
Internet Explorer browser.
Apache HTTPD access log:
131.177.146.160 [11/Jan/2010:12:42:49 +0200] "GET
/sites/one%2Bone%3Cthree HTTP/1.1" +302 -(-) 263 0
131.177.146.160 [11/Jan/2010:12:42:49 +0200] "GET
/contextroot/subcontext/sites/one+one%3cthree HTTP/1.1" +200
worker2(worker2) 399 0
Apache Tomcat access log ("standard" access valve):
131.177.146.160 - - [11/Jan/2010:12:44:59 +0200] "GET
/contextroot/subcontext/sites/one+one%3Cthree HTTP/1.1" 200 399
What my application actually sees after decoding the URL: sites/one one<three
Notice that mod_alias has erronously (considering the use case in
question) re-encoded the URL, causing %2B to change into '+' and %3C
to change into equivalent %3c.
Now, if I manually modify the address bar to access
http://localhost/contextroot/subcontext/sites/one%2Bone%3Cfour,
Apache HTTPD access log now shows:
131.177.146.160 [11/Jan/2010:12:53:37 +0200] "GET
/ts_core_virtual_repository/TeamCenterEmulator/sites/one%2Bone%3Cfour
HTTP/1.1" +200 worker1(worker1) 399 15625
but Tomcat access log still shows:
131.177.146.160 - - [11/Jan/2010:12:53:34 +0200] "GET
/ts_core_virtual_repository/TeamCenterEmulator/sites/one+one%3Cfour
HTTP/1.1" 200 399
and my application sees after decoding the URL: sites/one one<four
Quite interesting: No URL rewriting should occur at Apache HTTPD,
because the RedirectMatch rule does not match, but the URLs in HTTPD
and Tomcat access logs are semantically different.
OBSERVED BEHAVIOR WITH +ForwardURICompatUnparsed:
=============================================
Let's see what happens when I add to httpd.conf:
JkOptions +ForwardURICompatUnparsed
Let's try to address URL http://localhost/sites/one%2Bone%3Cfive with
Internet Explorer.
Apache HTTPD access log:
131.177.146.160 [11/Jan/2010:12:58:04 +0200] "GET
/sites/one%2Bone%3Cfive HTTP/1.1" +302 -(-) 278 0
131.177.146.160 [11/Jan/2010:12:58:04 +0200] "GET
/contextroot/subcontext/sites/one+one%3cfive HTTP/1.1" +200
worker1(worker1)
Apache Tomcat access log:
131.177.146.160 - - [11/Jan/2010:12:58:04 +0200] "GET
/ts_core_virtual_repository/TeamCenterEmulator/sites/one+one%3cfive
HTTP/1.1" 200 399
What my application actually sees after decoding: sites/one one<five
The behavior is similar to what was observed before.
Now, I again manually modify the address bar to access
http://localhost/contextroot/subcontext/sites/one%2Bone%3Csix. This
time,
Apache HTTPD access log shows:
131.177.146.160 [11/Jan/2010:13:00:20 +0200] "GET
/contextroot/subcontext/sites/one%2Bone%3csix HTTP/1.1" +200
worker1(worker1) 399 15625
and Tomcat access log shows:
131.177.146.160 - - [11/Jan/2010:13:00:18 +0200] "GET
/contextroot/subcontext/sites/one%2Bone%3csix HTTP/1.1" 200 399
This time my application sees the intended request after decoding the
URL: sites/one+one<six
THE ISSUE
=========
Both mod_alias and mod_jk seem to decode and re-encode the URL during
their processing, and it seems like once de-encoded, neither module
will re-encode the '+' character. Is this intentional and desirable?
JkOption ForwardURICompatUnparsed seem to offer a workaround for this
problem for the load balancer, but the solution has its own problems
(incompability with mod_rewrite among other things, although that does
not seem to apply to browser redirects). I have not found a similar
workaround for mod_alias yet.
(mod_rewrite does have the [B] option, "encode backreferences", but my
brief experiments with [B,R] failed miserably with the results being
"one%252bone%253csix" and similar double-encoded garbage).
How should I change my configuration so that
http://localhost/sites/one%2Bone%3Cthree gives the same same results
as http://localhost:8082/contextroot/subcontext/sites/one%2Bone%3Cthree?
Best Regards,
Tero Karttunen
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: mod_jk: plus-character causes %-encoding problems
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Mark,
On 1/13/2010 5:51 PM, Mark Thomas wrote:
> On 13/01/2010 22:47, Tero Karttunen wrote:
>> Thank you for another reply, Chris! I was secretly hoping that
>> somebody would stand up and tell me that I have missed something
>> obvious, but the more I look into this issue, the messier it seems.
>> But let's not get ahead of things.
>>
>> I apologize for the inconsistency in the log lines I posted in my
>> earlier message. I had tried to replace (and anonymize) the URL path
>> elements with s/ts_core_virtual_repository/contextroot/,
>> s/TeamCenterEmulator/subcontext/ and s/geek/localhost/, but I had
>> obviously missed some references. Sorry for the confusion. Even though
>> the cat's out of the bag, I will stick to the replacements for
>> consistency's sake.
>
> Just a thought, mod_jk doesn't always play nicely with other modules if
> those modules try to manipulate the URI. Have you tried mod_proxy_http?
mod_alias appears to be hosing the URL in this case, so I don't think
switching to mod_proxy_http would help.
On the other hand, simply switching from using mod_alias + mod_jk to
mod_proxy_http only would allow re-mapping of the URL and proxying all
in one step:
ProxyPass /sites http://localhost/contextroot/subcontext/sites
ProxyPassReverse /sites http://localhost/contextroot/subcontext/sites
Such a setup would avoid the trip through mod_alias and, potentially
avoid the broken URL.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAktOW5QACgkQ9CaO5/Lv0PCUkACeLvOwuqNzZT3cVHKb/aqdSbOU
QqgAnRDvZIPcA0egbdAhTQ81i9ICwFXb
=jwho
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: mod_jk: plus-character causes %-encoding problems
Posted by Mark Thomas <ma...@apache.org>.
On 13/01/2010 22:47, Tero Karttunen wrote:
> Thank you for another reply, Chris! I was secretly hoping that
> somebody would stand up and tell me that I have missed something
> obvious, but the more I look into this issue, the messier it seems.
> But let's not get ahead of things.
>
> I apologize for the inconsistency in the log lines I posted in my
> earlier message. I had tried to replace (and anonymize) the URL path
> elements with s/ts_core_virtual_repository/contextroot/,
> s/TeamCenterEmulator/subcontext/ and s/geek/localhost/, but I had
> obviously missed some references. Sorry for the confusion. Even though
> the cat's out of the bag, I will stick to the replacements for
> consistency's sake.
Just a thought, mod_jk doesn't always play nicely with other modules if
those modules try to manipulate the URI. Have you tried mod_proxy_http?
Mark
>
>>> Notice that mod_alias has erronously (considering the use case in
>>> question) re-encoded the URL, causing %2B to change into '+' and %3C
>>> to change into equivalent %3c.
>>
>> Note that mod_jk is not involved, here: mod_alias is performing the
>> redirect and mod_jk does not get involved. Also, the change from %3C to
>> %3c is not really a problem: HTTP allows either upper or lowercase
>> %-encoded URI elements (see section 2.2 of
>> http://www.ietf.org/rfc/rfc1738.txt).
>
> This is correct. The superceding RFC 3986 states: "If two URIs differ
> only in the case of hexadecimal digits used in percent-encoded octets,
> they are equivalent", but it also continues: "For consistency, URI
> producers and normalizers should use uppercase hexadecimal digits for
> all percent-encodings", making one choice preferable to the other.
>
> [JkOptions +ForwardUriProxy]
>>> Now, if I manually modify the address bar to access
>>> http://localhost/contextroot/subcontext/sites/one%2Bone%3Cfour,
>>> Apache HTTPD access log now shows:
>>> 131.177.146.160 [11/Jan/2010:12:53:37 +0200] "GET
>>> /ts_core_virtual_repository/TeamCenterEmulator/sites/one%2Bone%3Cfour
>>> HTTP/1.1" +200 worker1(worker1) 399 15625
>>
>> Good.
>>
>>> but Tomcat access log still shows:
>>> 131.177.146.160 - - [11/Jan/2010:12:53:34 +0200] "GET
>>> /ts_core_virtual_repository/TeamCenterEmulator/sites/one+one%3Cfour
>>> HTTP/1.1" 200 399
>>
>> Right: that's wrong.
>>
>>> and my application sees after decoding the URL: sites/one one<four
>>
>> Given that Tomcat saw one+one%3Cfour, this is correct decoding.
>>
>> What does the mod_jk log show for this request?
>
> I reran the test with JK request logging on, and the log shows:
> [Wed Jan 13 12:14:13 2010] worker1 GET
> /contextroot/subcontext/sites/one%2Bone%3Cfour HTTP/1.1 200 0.000000
>
> so both Apache HTTPD and mod_jk logs show the correctly encoded URLs
> from the browser request.
>
>>> Quite interesting: No URL rewriting should occur at Apache HTTPD,
>>> because the RedirectMatch rule does not match, but the URLs in HTTPD
>>> and Tomcat access logs are semantically different.
>>
>> Well, the RedirectMatch rule does match for the first request, and it
>> definitely appears that mod_alias is mangling your URL. Have you tried
>> snooping the HTTP conversation to make sure it's not your web browser
>> that is misinterpreting the 302 response from httpd?
>
> Yes I have. Here is a telnet session for proof.
>
> ==============================
> <username>@<hostname>:~ $ telnet localhost 80
> Trying 131.177.146.160...
> Connected to <hostname>.
> Escape character is '^]'.
> GET /sites/one%2Bone%3C HTTP/1.0
>
> HTTP/1.1 302 Found
> Date: Wed, 13 Jan 2010 09:57:32 GMT
> Server: Apache/2.2.14 (Win32) mod_jk/1.2.28
> Location: http://localhost/contextroot/subcontext/sites/one+one%3c
> Content-Length: 274
> Connection: close
> Content-Type: text/html; charset=iso-8859-1
>
> <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
> <html><head>
> <title>302 Found</title>
> </head><body>
> <h1>Found</h1>
> <p>The document has moved <a
> href="http://localhost/contextroot/subcontext/sites/one+one%3c">here</a>.</p>
> </body></html>
> ==============================
>
> It looks to me like both mod_alias and mod_jk (with +ForwardUriProxy)
> decode the URLs and do not subsequently re-encode the '+' character.
>
> I do not have any C coding experience, but I attempted to check mod_jk
> source code for this. mod_jk uses int jk_canonenc(const char *x, char
> *y, int maxlen) function for encoding when +ForwardUriProxy is on.
> Here are the juicy bits (slightly reformatted for brevity):
>
> /* characters which should not be encoded */
> char *allowed = "~$-_.+!*'(),;:@&=";
> /* characters which much not be en/de-coded */
> char *reserved = "/";
> for (i = 0, j = 0; ch != '\0' && j < maxlen; i++, j++, ch=x[i]) {
> if (strchr(reserved, ch)) { /* always handle '/' first */
> y[j] = ch;
> continue;
> }
> if (!JK_ISALNUM(ch) && !strchr(allowed, ch)) { /* recode it,
> if necessary */
> if (j+2<maxlen) {
> jk_c2hex(ch, &y[j]);
> j += 2;
> }
> else {
> return JK_FALSE;
> }
> }
> else {
> y[j] = ch;
> }
> }
>
> mod_alias and mod_rewrite are already reported to suffer from similar
> encoding problems. There are several bug reports; the best I could
> find was https://issues.apache.org/bugzilla/show_bug.cgi?id=32328.
> Link http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=421820 is also
> useful in trying to find out what is going on, as is RFC 3986.
>
> All this is making my head hurt, but what I guess is going on is that
> the original URL (still available as r->unparsed_uri) is being decoded
> in Apache HTTPD at a very early stage, and once mod_jk or other
> dispatchers activate, the r->uri they handle can already be a result
> of multiple URI manipulations by mod_rewrite and other modules, and
> for that reason it can be considered unsafe to mindlessly re-encode
> some of its reserved characters. But this is only my first guess.
>
> +ForwardUriCompatUnparsed solves the mod_jk part of the problem _for
> me_, but while HTTPD people are working on bug 32328 (since 2007),
> could it be benecifial for mod_jk to maybe offer a fifth Forwarding
> mode as a workaround for the problem for mod_jk users? Maybe taking a
> list of characters to be encoded as an extra argument?
>
> Unfortunately, I still have no ideas on how to configure the URL
> redirection for Apache HTTPD so that the plus-characters are preserved
> in encoded format. Does anyone have any ideas or hints?
>
> Thanks for help!
>
> Tero Karttunen
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: mod_jk: plus-character causes %-encoding problems
Posted by Tero Karttunen <ka...@gmail.com>.
Thank you for another reply, Chris! I was secretly hoping that
somebody would stand up and tell me that I have missed something
obvious, but the more I look into this issue, the messier it seems.
But let's not get ahead of things.
I apologize for the inconsistency in the log lines I posted in my
earlier message. I had tried to replace (and anonymize) the URL path
elements with s/ts_core_virtual_repository/contextroot/,
s/TeamCenterEmulator/subcontext/ and s/geek/localhost/, but I had
obviously missed some references. Sorry for the confusion. Even though
the cat's out of the bag, I will stick to the replacements for
consistency's sake.
>> Notice that mod_alias has erronously (considering the use case in
>> question) re-encoded the URL, causing %2B to change into '+' and %3C
>> to change into equivalent %3c.
>
> Note that mod_jk is not involved, here: mod_alias is performing the
> redirect and mod_jk does not get involved. Also, the change from %3C to
> %3c is not really a problem: HTTP allows either upper or lowercase
> %-encoded URI elements (see section 2.2 of
> http://www.ietf.org/rfc/rfc1738.txt).
This is correct. The superceding RFC 3986 states: "If two URIs differ
only in the case of hexadecimal digits used in percent-encoded octets,
they are equivalent", but it also continues: "For consistency, URI
producers and normalizers should use uppercase hexadecimal digits for
all percent-encodings", making one choice preferable to the other.
[JkOptions +ForwardUriProxy]
>> Now, if I manually modify the address bar to access
>> http://localhost/contextroot/subcontext/sites/one%2Bone%3Cfour,
>> Apache HTTPD access log now shows:
>> 131.177.146.160 [11/Jan/2010:12:53:37 +0200] "GET
>> /ts_core_virtual_repository/TeamCenterEmulator/sites/one%2Bone%3Cfour
>> HTTP/1.1" +200 worker1(worker1) 399 15625
>
> Good.
>
>> but Tomcat access log still shows:
>> 131.177.146.160 - - [11/Jan/2010:12:53:34 +0200] "GET
>> /ts_core_virtual_repository/TeamCenterEmulator/sites/one+one%3Cfour
>> HTTP/1.1" 200 399
>
> Right: that's wrong.
>
>> and my application sees after decoding the URL: sites/one one<four
>
> Given that Tomcat saw one+one%3Cfour, this is correct decoding.
>
> What does the mod_jk log show for this request?
I reran the test with JK request logging on, and the log shows:
[Wed Jan 13 12:14:13 2010] worker1 GET
/contextroot/subcontext/sites/one%2Bone%3Cfour HTTP/1.1 200 0.000000
so both Apache HTTPD and mod_jk logs show the correctly encoded URLs
from the browser request.
>> Quite interesting: No URL rewriting should occur at Apache HTTPD,
>> because the RedirectMatch rule does not match, but the URLs in HTTPD
>> and Tomcat access logs are semantically different.
>
> Well, the RedirectMatch rule does match for the first request, and it
> definitely appears that mod_alias is mangling your URL. Have you tried
> snooping the HTTP conversation to make sure it's not your web browser
> that is misinterpreting the 302 response from httpd?
Yes I have. Here is a telnet session for proof.
==============================
<username>@<hostname>:~ $ telnet localhost 80
Trying 131.177.146.160...
Connected to <hostname>.
Escape character is '^]'.
GET /sites/one%2Bone%3C HTTP/1.0
HTTP/1.1 302 Found
Date: Wed, 13 Jan 2010 09:57:32 GMT
Server: Apache/2.2.14 (Win32) mod_jk/1.2.28
Location: http://localhost/contextroot/subcontext/sites/one+one%3c
Content-Length: 274
Connection: close
Content-Type: text/html; charset=iso-8859-1
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>302 Found</title>
</head><body>
<h1>Found</h1>
<p>The document has moved <a
href="http://localhost/contextroot/subcontext/sites/one+one%3c">here</a>.</p>
</body></html>
==============================
It looks to me like both mod_alias and mod_jk (with +ForwardUriProxy)
decode the URLs and do not subsequently re-encode the '+' character.
I do not have any C coding experience, but I attempted to check mod_jk
source code for this. mod_jk uses int jk_canonenc(const char *x, char
*y, int maxlen) function for encoding when +ForwardUriProxy is on.
Here are the juicy bits (slightly reformatted for brevity):
/* characters which should not be encoded */
char *allowed = "~$-_.+!*'(),;:@&=";
/* characters which much not be en/de-coded */
char *reserved = "/";
for (i = 0, j = 0; ch != '\0' && j < maxlen; i++, j++, ch=x[i]) {
if (strchr(reserved, ch)) { /* always handle '/' first */
y[j] = ch;
continue;
}
if (!JK_ISALNUM(ch) && !strchr(allowed, ch)) { /* recode it,
if necessary */
if (j+2<maxlen) {
jk_c2hex(ch, &y[j]);
j += 2;
}
else {
return JK_FALSE;
}
}
else {
y[j] = ch;
}
}
mod_alias and mod_rewrite are already reported to suffer from similar
encoding problems. There are several bug reports; the best I could
find was https://issues.apache.org/bugzilla/show_bug.cgi?id=32328.
Link http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=421820 is also
useful in trying to find out what is going on, as is RFC 3986.
All this is making my head hurt, but what I guess is going on is that
the original URL (still available as r->unparsed_uri) is being decoded
in Apache HTTPD at a very early stage, and once mod_jk or other
dispatchers activate, the r->uri they handle can already be a result
of multiple URI manipulations by mod_rewrite and other modules, and
for that reason it can be considered unsafe to mindlessly re-encode
some of its reserved characters. But this is only my first guess.
+ForwardUriCompatUnparsed solves the mod_jk part of the problem _for
me_, but while HTTPD people are working on bug 32328 (since 2007),
could it be benecifial for mod_jk to maybe offer a fifth Forwarding
mode as a workaround for the problem for mod_jk users? Maybe taking a
list of characters to be encoded as an extra argument?
Unfortunately, I still have no ideas on how to configure the URL
redirection for Apache HTTPD so that the plus-characters are preserved
in encoded format. Does anyone have any ideas or hints?
Thanks for help!
Tero Karttunen
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: mod_jk: plus-character causes %-encoding problems
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Tero,
On 1/11/2010 8:41 AM, Tero Karttunen wrote:
>
[snip]
> RedirectMatch ^/sites(.*)$ /contextroot/subcontext/sites$1
[snip]
> OBSERVED BEHAVIOR WITH +ForwardURIProxy: (implicit)
> =====================================
> I try to access URL http://localhost/sites/one%2Bone%3Cthree with
> Internet Explorer browser.
> Apache HTTPD access log:
> 131.177.146.160 [11/Jan/2010:12:42:49 +0200] "GET
> /sites/one%2Bone%3Cthree HTTP/1.1" +302 -(-) 263 0
> 131.177.146.160 [11/Jan/2010:12:42:49 +0200] "GET
> /contextroot/subcontext/sites/one+one%3cthree HTTP/1.1" +200
> worker2(worker2) 399 0
As you surmised, the RedirectMatch appears to be breaking your URI: %2B
- -> + instead of %2B -> %2B.
> Notice that mod_alias has erronously (considering the use case in
> question) re-encoded the URL, causing %2B to change into '+' and %3C
> to change into equivalent %3c.
Note that mod_jk is not involved, here: mod_alias is performing the
redirect and mod_jk does not get involved. Also, the change from %3C to
%3c is not really a problem: HTTP allows either upper or lowercase
%-encoded URI elements (see section 2.2 of
http://www.ietf.org/rfc/rfc1738.txt).
> Now, if I manually modify the address bar to access
> http://localhost/contextroot/subcontext/sites/one%2Bone%3Cfour,
> Apache HTTPD access log now shows:
> 131.177.146.160 [11/Jan/2010:12:53:37 +0200] "GET
> /ts_core_virtual_repository/TeamCenterEmulator/sites/one%2Bone%3Cfour
> HTTP/1.1" +200 worker1(worker1) 399 15625
Good.
> but Tomcat access log still shows:
> 131.177.146.160 - - [11/Jan/2010:12:53:34 +0200] "GET
> /ts_core_virtual_repository/TeamCenterEmulator/sites/one+one%3Cfour
> HTTP/1.1" 200 399
Right: that's wrong.
> and my application sees after decoding the URL: sites/one one<four
Given that Tomcat saw one+one%3Cfour, this is correct decoding.
What does the mod_jk log show for this request?
> Quite interesting: No URL rewriting should occur at Apache HTTPD,
> because the RedirectMatch rule does not match, but the URLs in HTTPD
> and Tomcat access logs are semantically different.
Well, the RedirectMatch rule does match for the first request, and it
definitely appears that mod_alias is mangling your URL. Have you tried
snooping the HTTP conversation to make sure it's not your web browser
that is misinterpreting the 302 response from httpd?
> Now, I again manually modify the address bar to access
> http://localhost/contextroot/subcontext/sites/one%2Bone%3Csix. This
> time,
> Apache HTTPD access log shows:
> 131.177.146.160 [11/Jan/2010:13:00:20 +0200] "GET
> /contextroot/subcontext/sites/one%2Bone%3csix HTTP/1.1" +200
> worker1(worker1) 399 15625
> and Tomcat access log shows:
> 131.177.146.160 - - [11/Jan/2010:13:00:18 +0200] "GET
> /contextroot/subcontext/sites/one%2Bone%3csix HTTP/1.1" 200 399
> This time my application sees the intended request after decoding the
> URL: sites/one+one<six
This all looks correct. When your web browser sends a correct URI, your
webapp sees the correct URI. I would leave +ForwardURICompatUnparsed
enabled for mod_jk.
> JkOption ForwardURICompatUnparsed seem to offer a workaround for this
> problem for the load balancer, but the solution has its own problems
> (incompability with mod_rewrite among other things, although that does
> not seem to apply to browser redirects). I have not found a similar
> workaround for mod_alias yet.
You aren't using mod_rewrite, so such incompatibilities should not be a
problem.
> How should I change my configuration so that
> http://localhost/sites/one%2Bone%3Cthree gives the same same results
> as http://localhost:8082/contextroot/subcontext/sites/one%2Bone%3Cthree?
- From your observations, I would expect that this is either a bug in
mod_alias (%2B -> +) or in your web browser. Only reading the HTTP
traffic will tell you which it is.
Good luck,
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAktM4VQACgkQ9CaO5/Lv0PCdhACeIHMjqzMTxiOLC7h8fv7bnTl7
j6AAoI5h0SSglg1zfQ3588H4J8Ey8PsD
=Ike1
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: mod_jk: plus-character causes %-encoding problems
Posted by Tero Karttunen <ka...@gmail.com>.
> Is UTF-8 the reason why you are using your custom decoding?
> [...]
> You should be able to use HttpServletRequest.getPathInfo() to get the
> decoded value.
Not really. I could probably be using getPathInfo() for getting the
decoded request. But note that I am also decoding both the former urls
from the CSV input file and comparing the result with the decoded
request, so in any case I have to use some custom decoder in my code.
In these kinds of situations, I reflexively do the decoding with the
same decoder to make absolutely sure that the decoded strings match
for identical inputs. Otherwise if I let the servlet container use its
decoder for requests and I use my own custom decoder to decode the
mapping table, I may run into trouble unnecessary and - in the worst
case - make my application application server-specific. Of course,
such a thing could never happen with Tomcat! :-)
Thanks for pointing this out!
Tero Karttunen
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: mod_jk: plus-character causes %-encoding problems
Posted by Konstantin Kolinko <kn...@gmail.com>.
2010/1/14 Tero Karttunen <ka...@gmail.com>:
>> Why is '+' decoded to ' ' in the path part of the URL?
>> That is, I think, wrong.
>
> This is an interesting theory. If true, it could provide an
> explanation to the observed behavior, but I cannot completely follow
> it.
>
>> The '+' char has no special meaning in HTTP/1.1 (RFC 2616) [1], so in
>> the path part of the URL it just means itself, the plus sign.
>
> On the other hand, the same RFC provides a counter-example. Look at
> section 3.2.3 "URI comparison". It says that characters other than
> those in the "reserved" and "unsafe" sets are equivalent to their
> %-encoded counterparts. The reserved set as defined in RFC 2396 (and
> the later RFC 3986 that obsoletes it) include '+' character.
>
> I believe the chapter 3.2.3 means that the characters in the reserved
> set are not equivalent to their %-encoded counterparts, and in this
> way, /contextroot/subcontext/sites/one+one%3cfive IS NOT equivalent to
> /contextroot/subcontext/sites/one%2Bone%3cfive when doing URI
> comparison.
>
>> It is the HTML Forms spec [2] that makes it special, defining
>> "urlencoding" used when submitting web forms through HTTP. It has
>> special meaning only in the query part of the URL and only because of
>> that part of HTML spec.
>
> HTML Forms spec does define www-form-urlencoding, but I can't tell
> from the spec whether it is limited to just the query part.
>
>>> What my application actually sees after decoding: sites/one one<five
>>
>> What is your application code here? Where and how do you obtain the
>> "decoded" value?
>
> I am using Apache Commons URLCodec to decode the URL. This widely-used
> utility class does not make the distinction between path and query
> parts...
>
> Let me explain my application to you before I provide the code example
> to you. As you could guess from its name TeamCenterEmulator, my
> application emulates a set of former URLs, continuing to serve the
> pre-existing links while the legacy application is retired.
>
> My application is configured with a CSV file containing a mapping
> between an URL and a resource it is supposed to serve (in a dynamic
> fashion, it is not a simple file). Say, the application could contain
> the following mapping:
>
> <former url> <response>
> /sites/foo file1
> /sites/bar file2
> /sites/one%2Bone%cthree file3
> /sites/one%2Bone%cfour file4
> /sites/one%2Bone%cfive file5
> ...
>
> Once the application initializes, it reads the mapping into memory,
> and if the request matches the former url EXACTLY, the matching
> response is returned. This is the application spec. Note here that by
> RFC 2616-compliant URI comparison, my application must regard request
> /sites/one+one%cfive as a non-match!
>
> Here is doGet from my servlet. Note that I am trimming the URL to
> start from the "sites" part for obvious reasons...
>
> protected void doGet(HttpServletRequest request, HttpServletResponse
> response) throws ServletException, IOException {
> super.doGet(request, response);
> if (config == null) {
> config = new
> ConfigurationFactory().createConfiguration(getServletContext().getInitParame
> ter("teamCenterURLMapping"));
> }
> String urlSnippet = (getServletContext().getContextPath() +
> "/" + getServletConfig().getServletName() + "/");
> String url = "";
> if (request.getRequestURI().length() > urlSnippet.length())
> {
> url =
> request.getRequestURI().substring(urlSnippet.length());
> }
> try {
> TeamCenterConfigurationItem item =
> config.findByURL(url);
> [...]
> catch (UnknownUrlException) {
> ...
> }
> }
>
> I am not going to post ConfigurationFactory, because it is not
> interesting. It basically builds a HashMap based on the CSV file that
> has URLCodec.decode()'d former urls as its keys, with the idea that if
> we URL-decode the incoming request, we can search the HashMap for
> matches.
>
> Here is how the abovementioned findUrl method does just that:
>
> public TeamCenterConfigurationItem findByURL (String url) throws
> UnknownUrlException {
> URLCodec codec = new URLCodec("UTF8");
> try {
> url = codec.decode(url);
> logger.info(url);
> } catch (DecoderException e) {
> logger.error(e);
> throw new UnknownUrlException (url);
> }
> if (config.containsKey(url)) {
> return config.get(url);
> }
> throw new UnknownUrlException (url);
> }
>
> What do you think? Is my approach valid? Am I somehow abusing
> URLCodec? Should the request be (partially) decoded in some other way?
>
> Best Regards,
> Tero Karttunen
>
Is UTF-8 the reason why you are using your custom decoding?
There is URIEncoding on a <Connector> element [1] and a usual
(non-default) setting for it is URIEncoding="UTF-8". There is a FAQ
page about solving character encoding issues [2].
You should be able to use HttpServletRequest.getPathInfo() to get the
decoded value.
[1]
http://tomcat.apache.org/tomcat-6.0-doc/config/http.html
http://tomcat.apache.org/tomcat-6.0-doc/config/ajp.html
[2]
http://wiki.apache.org/tomcat/FAQ/CharacterEncoding
Best regards,
Konstantin Kolinko
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: mod_jk: plus-character causes %-encoding problems
Posted by Tero Karttunen <ka...@gmail.com>.
> Why is '+' decoded to ' ' in the path part of the URL?
> That is, I think, wrong.
This is an interesting theory. If true, it could provide an
explanation to the observed behavior, but I cannot completely follow
it.
> The '+' char has no special meaning in HTTP/1.1 (RFC 2616) [1], so in
> the path part of the URL it just means itself, the plus sign.
On the other hand, the same RFC provides a counter-example. Look at
section 3.2.3 "URI comparison". It says that characters other than
those in the "reserved" and "unsafe" sets are equivalent to their
%-encoded counterparts. The reserved set as defined in RFC 2396 (and
the later RFC 3986 that obsoletes it) include '+' character.
I believe the chapter 3.2.3 means that the characters in the reserved
set are not equivalent to their %-encoded counterparts, and in this
way, /contextroot/subcontext/sites/one+one%3cfive IS NOT equivalent to
/contextroot/subcontext/sites/one%2Bone%3cfive when doing URI
comparison.
> It is the HTML Forms spec [2] that makes it special, defining
> "urlencoding" used when submitting web forms through HTTP. It has
> special meaning only in the query part of the URL and only because of
> that part of HTML spec.
HTML Forms spec does define www-form-urlencoding, but I can't tell
from the spec whether it is limited to just the query part.
>> What my application actually sees after decoding: sites/one one<five
>
> What is your application code here? Where and how do you obtain the
> "decoded" value?
I am using Apache Commons URLCodec to decode the URL. This widely-used
utility class does not make the distinction between path and query
parts...
Let me explain my application to you before I provide the code example
to you. As you could guess from its name TeamCenterEmulator, my
application emulates a set of former URLs, continuing to serve the
pre-existing links while the legacy application is retired.
My application is configured with a CSV file containing a mapping
between an URL and a resource it is supposed to serve (in a dynamic
fashion, it is not a simple file). Say, the application could contain
the following mapping:
<former url> <response>
/sites/foo file1
/sites/bar file2
/sites/one%2Bone%cthree file3
/sites/one%2Bone%cfour file4
/sites/one%2Bone%cfive file5
...
Once the application initializes, it reads the mapping into memory,
and if the request matches the former url EXACTLY, the matching
response is returned. This is the application spec. Note here that by
RFC 2616-compliant URI comparison, my application must regard request
/sites/one+one%cfive as a non-match!
Here is doGet from my servlet. Note that I am trimming the URL to
start from the "sites" part for obvious reasons...
protected void doGet(HttpServletRequest request, HttpServletResponse
response) throws ServletException, IOException {
super.doGet(request, response);
if (config == null) {
config = new
ConfigurationFactory().createConfiguration(getServletContext().getInitParame
ter("teamCenterURLMapping"));
}
String urlSnippet = (getServletContext().getContextPath() +
"/" + getServletConfig().getServletName() + "/");
String url = "";
if (request.getRequestURI().length() > urlSnippet.length())
{
url =
request.getRequestURI().substring(urlSnippet.length());
}
try {
TeamCenterConfigurationItem item =
config.findByURL(url);
[...]
catch (UnknownUrlException) {
...
}
}
I am not going to post ConfigurationFactory, because it is not
interesting. It basically builds a HashMap based on the CSV file that
has URLCodec.decode()'d former urls as its keys, with the idea that if
we URL-decode the incoming request, we can search the HashMap for
matches.
Here is how the abovementioned findUrl method does just that:
public TeamCenterConfigurationItem findByURL (String url) throws
UnknownUrlException {
URLCodec codec = new URLCodec("UTF8");
try {
url = codec.decode(url);
logger.info(url);
} catch (DecoderException e) {
logger.error(e);
throw new UnknownUrlException (url);
}
if (config.containsKey(url)) {
return config.get(url);
}
throw new UnknownUrlException (url);
}
What do you think? Is my approach valid? Am I somehow abusing
URLCodec? Should the request be (partially) decoded in some other way?
Best Regards,
Tero Karttunen
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: mod_jk: plus-character causes %-encoding problems
Posted by Tero Karttunen <ka...@gmail.com>.
>> The '+' char has no special meaning in HTTP/1.1 (RFC 2616) [1], so in
>> the path part of the URL it just means itself, the plus sign.
>
> Any bug in either mod_alias or mod_jk could be proven with regard to the
> above statement by simply changing the URL from:
>
> http://localhost/sites/one%2Bone%3Cthree
> to:
> http://localhost/sites?one%2Bone%3Cthree
Whaddya know - the test results show that Konstantin is right! The
query mark makes all the difference, and incidently, '+' seems to be
the only character that is being treated differently by the decoder.
When presented with a query
http://localhost/sites/one%2Bone%3Cthree?one%2Bone%3Cthree
mod_alias responds with a redirect to
http://localhost/contextroot/subcontext/sites/one+one%3cthree?one%2Bone%3Cthree
,
leaving the query part intact.
mod_jk also behaves in a similar fashion with +ForwardURIProxy. A request to
http://localhost/contextroot/subcontext/sites/one%2Bone%3Cthree?one%2Bone%3Cthree
shows up as
http://localhost/contextroot/subcontext/sites/one+one%3Cthree?one%2Bone%3Cthree
in Tomcat's logs.
Furthermore org.apache.tomcat.util.buf.UDecoder and
org.apache.catalina.util.RequestUtil classes seem to have special
handling logic for query parts which enable this kind of checks:
if (b == '+' && isQuery) {
b = (byte)' ';
} else if (b == '%') {
b = (byte) ((convertHexDigit(bytes[ix++]) << 4)
+ convertHexDigit(bytes[ix++]));
}
I have not yet actually tested this in practice, but look like the
facts speak for themselves. I still cannot find this functionality in
specs, but it looks like I don't have to. :-) Good job, Konstantin!
In the end, the only real bug here seems to reside my application's
use of Apache Commons URLCodec. I will have to replace that with some
other solution that respects the difference between path and query
parts. I should be able to use +ForwardURIProxy as well. It may be a
different solution than I expected and it does not make a difference
with %2B and '+' characters, but it should work and be spec- (or at
least reality-) compliant. :-)
I will let you know whether it solves the problem.
Tero Karttunen
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: mod_jk: plus-character causes %-encoding problems
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Konstantin,
On 1/13/2010 6:42 PM, Konstantin Kolinko wrote:
> Why is '+' decoded to ' ' in the path part of the URL?
>
> That is, I think, wrong.
>
> The '+' char has no special meaning in HTTP/1.1 (RFC 2616) [1], so in
> the path part of the URL it just means itself, the plus sign.
Any bug in either mod_alias or mod_jk could be proven with regard to the
above statement by simply changing the URL from:
http://localhost/sites/one%2Bone%3Cthree
to:
http://localhost/sites?one%2Bone%3Cthree
...and see what happens.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAktQfFAACgkQ9CaO5/Lv0PAwxwCfbjmJDLFr7NLDUEXF6UYkOuH5
/fAAnith23hNQ91dV9D2zmB6GZeVv5oE
=e+uu
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: mod_jk: plus-character causes %-encoding problems
Posted by Konstantin Kolinko <kn...@gmail.com>.
2010/1/11 Tero Karttunen <ka...@gmail.com>:
> Apache Tomcat access log:
> 131.177.146.160 - - [11/Jan/2010:12:58:04 +0200] "GET
> /ts_core_virtual_repository/TeamCenterEmulator/sites/one+one%3cfive
> HTTP/1.1" 200 399
> What my application actually sees after decoding: sites/one one<five
Why is '+' decoded to ' ' in the path part of the URL?
That is, I think, wrong.
The '+' char has no special meaning in HTTP/1.1 (RFC 2616) [1], so in
the path part of the URL it just means itself, the plus sign.
It is the HTML Forms spec [2] that makes it special, defining
"urlencoding" used when submitting web forms through HTTP. It has
special meaning only in the query part of the URL and only because of
that part of HTML spec.
[1] http://tools.ietf.org/html/rfc2616
[2] http://www.w3.org/TR/html401/interact/forms.html#h-17.13.4.1
> What my application actually sees after decoding: sites/one one<five
What is your application code here? Where and how do you obtain the
"decoded" value?
E.g. calling
http://localhost:8080/examples/servlets/servlet/RequestInfoExample/foo+bar
in 6.0.20 and also in 6.0.22 RC displays
Path Info: /foo+bar
so that API is working properly.
Best regards,
Konstantin Kolinko
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org