request.getPathInfo() gets truncated when ";" is present

["Peter J. Farrell" <pe...@mach-ii.com>]

I'm trying to figure out a particular strange problem with 
request.getPathInfo() data being truncated.  This has to do with SES / 
Friendly URLs in which data is append on the path info part of the 
request URI.  On the framework project I work on, we've seen developers 
put data into SES URL -- in order to keep the URI compliant for web 
servers we URL encode special characters (i.e. & -> &).  As you may 
know, anything can be game in URLs theses days.

Anyways, the problem we're seeing with Tomcat is that it incorrectly 
parses the pathInfo when a ";" is present.  Since an example is worth a 
million words, here is example request URI (yes, CFML engine here 
deployed as a WAR):

/index.cfm/somePathInfo&withMoreInfo/

One would expect the path info data to be:

/somePathInfo&withMoreInfo/

However Tomcat truncates everything after start the ";".  The end result 
when calling request.getPathInfo() is:

/somePathInfo&amp

I know that Apache is passing the correct information to Tomcat because 
request.getRequestURI() returns the full path including the complete 
path info. The problem is this breaks the path info because it is not 
the path info has been truncated.  There is nothing in the servlet specs 
that I can find that says that ";" cannot be in the path info and it 
seems valid according to the RFCs available that ";" in path info is a 
valid character.

I've done a bit of poking around in the Tomcat code base, but nothing 
jumps out at me at where this going wrong (although I could be looking 
in the wrong places).  Does anybody know where this is going wrong?  If 
so I'd like to know so I can file ticket regarding this problem.

.Peter

Re: request.getPathInfo() gets truncated when ";" is present

[Christopher Schultz <ch...@christopherschultz.net>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Peter,

On 7/24/2009 8:18 PM, maestrofjp wrote:
>> Since our framework is deployed on several different CFML servlets --
>> their implementation to get at the original http request wrapper differs
>> a bit (three different vendors).  We'll probably stick to use the poor
>> man's encoding using a modified unicode representation of ";" in the
>> end.  Another solution is to write a filter and use the getRequestURI()
>> and replace the bad path info in the request with the full length version.
> 
>> You could implement a Filter that wraps the request for these servlets.
>> The wrapped request would override the getPathInfo method and perform
>> the reverse-engineering of the path info you describe above.
> 
> Yeah, the problem is the CFML engines run on a multitude of containers (not
> just Tomcat) and I'm not a contributor to any of the open source engines at
> the moment.

That shouldn't matter: the filter should not perform differently under
other containers. If you're worried about performance, you could have
the filter initialize itself and check the version of the container and
turn itself into a no-op when it's not Tomcat running.

>> This would allow you to "patch" your servlets all at once without
>> actually modifying their code, and without going through the backflips
>> of an alternate URI encoding.
> 
> One engine is proprietary engine by Adobe -- it's been like pulling teeth to
> get them to look into the issue.  I doubt they will change their engine to
> fix this defect.

That's why I'm suggesting that you patch it from the outside: the code
is easy to write and does not require you to modify anything inside
their product.

> Yeah, I'll submit a bug report and see what happens -- I'll definitely
> submit a patch if I can find an elegant solution to it.  i'm pretty
> unfamiliar with the Tomcat code base itself.  Someone did mention about
> having to recombine stuff post path resolution -- I'm not sure exactly what
> was mention since I'm unfamiliar with the internals of Tomcat.

Start with the code for the HTTP connector
(java/org/apache/coyote/http11/Http11Processor.java). The process()
method is where the magic essentially begins.

- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEUEARECAAYFAkptu9YACgkQ9CaO5/Lv0PDFVwCXfWl7hmf0mQz4fmEMWXM1SDGP
SQCfRIBrZ+Cl0dVYKpbMBECj0iwTsSA=
=cgCA
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: request.getPathInfo() gets truncated when ";" is present

[maestrofjp <pe...@mach-ii.com>]

> Since our framework is deployed on several different CFML servlets --
> their implementation to get at the original http request wrapper differs
> a bit (three different vendors).  We'll probably stick to use the poor
> man's encoding using a modified unicode representation of ";" in the
> end.  Another solution is to write a filter and use the getRequestURI()
> and replace the bad path info in the request with the full length version.

>You could implement a Filter that wraps the request for these servlets.
>The wrapped request would override the getPathInfo method and perform
>the reverse-engineering of the path info you describe above.

Yeah, the problem is the CFML engines run on a multitude of containers (not
just Tomcat) and I'm not a contributor to any of the open source engines at
the moment.

>This would allow you to "patch" your servlets all at once without
>actually modifying their code, and without going through the backflips
>of an alternate URI encoding.

One engine is proprietary engine by Adobe -- it's been like pulling teeth to
get them to look into the issue.  I doubt they will change their engine to
fix this defect.

Another engine Open BlueDragon ships with an SES Url Filter.  It doesn't
suffer the same truncation problem as the Adobe engine because it's doing
what you suggested by bypassing the path info tomcat provides and parses
throught the request.getRequestURI() itself and builds the correct path info
and sets that to request.

I'm unsure about the behaviour of the open source Railo engine because we
just haven't had the time to verify it on that specific engine as such.  I
suspect that it suffers from the ";" truncation problem. 

>You could also log a bug and see what happens. As Bill points out,
>nobody has been motivated to fix this issue. If you log a bug and get
>some feedback, perhaps you could fix it yourself and submit a patch (if
>you're motivated). Or, maybe some other kind soul will submit a patch
>for it. Since you are motivated to have it work for your application,
>you are a good candidate to fix it :)

Yeah, I'll submit a bug report and see what happens -- I'll definitely
submit a patch if I can find an elegant solution to it.  i'm pretty
unfamiliar with the Tomcat code base itself.  Someone did mention about
having to recombine stuff post path resolution -- I'm not sure exactly what
was mention since I'm unfamiliar with the internals of Tomcat.

Thanks for you help,
Peter
-- 
View this message in context: http://www.nabble.com/request.getPathInfo%28%29-gets-truncated-when-%22-%22-is-present-tp24633051p24653695.html
Sent from the Tomcat - User mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: request.getPathInfo() gets truncated when ";" is present

[Christopher Schultz <ch...@christopherschultz.net>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Peter,

On 7/24/2009 3:58 AM, Peter J. Farrell wrote:
> @Bill, thanks for mentioning the specific RFC referencing the encoding
> of ";".  That's what the team here figured out as well and it's nice to
> have an independent person verify our assumptions after reading the RFC
> sections.  Is it safe to assume that Tomcat code base assumes that
> anything after the ";" has to be the jsessionid?  That's our assumption
> at the moment.  Yes, there is a work around by using the
> request.getRequestURI()  instead as that has the whole URI and manually
> remove the "absolute" path to get the complete path info.
> 
> Since our framework is deployed on several different CFML servlets --
> their implementation to get at the original http request wrapper differs
> a bit (three different vendors).  We'll probably stick to use the poor
> man's encoding using a modified unicode representation of ";" in the
> end.  Another solution is to write a filter and use the getRequestURI()
> and replace the bad path info in the request with the full length version.

You could implement a Filter that wraps the request for these servlets.
The wrapped request would override the getPathInfo method and perform
the reverse-engineering of the path info you describe above.

This would allow you to "patch" your servlets all at once without
actually modifying their code, and without going through the backflips
of an alternate URI encoding.

You could also log a bug and see what happens. As Bill points out,
nobody has been motivated to fix this issue. If you log a bug and get
some feedback, perhaps you could fix it yourself and submit a patch (if
you're motivated). Or, maybe some other kind soul will submit a patch
for it. Since you are motivated to have it work for your application,
you are a good candidate to fix it :)

- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkpqCj4ACgkQ9CaO5/Lv0PDBTQCfVb01DUBJmtCSyamIWgaGEZ1F
ab8An2uVpIy8L4gxpvqTDM5XRN0soWcn
=JC+6
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: request.getPathInfo() gets truncated when ";" is present

["Peter J. Farrell" <pe...@mach-ii.com>]

@All, thanks everybody for responding so far. I apologize for not 
originally including the version of Tomcat we're using (6.0.18).  It was 
an oversight on my part in my hurried effort to write the email -- 
totally just blitzed that I should include that (my first post to the 
Tomcat list).

I realize that I accidentally posted an incorrect example.  The question 
posted by Christopher S. using the URLEncoded value of %3B is the 
correct encoding.  I shouldn't have pasted in a plain text ";".

This ";" is an edge case for us.  This URL encoding is part of a SES / 
Friendly URL implementation of a framework I'm a contributor for and 
therefore this was logged as defect report by a framework user since 
originally we were not URL encoding special characters (they were 
including a short "error" message in the URL proper).  We have no clue 
what people are generating and how they using as URLs -- just trying to 
get an idea of why this wasn't working.

@Christopher S., Sadly Tomcat still truncates the path info even when 
encoding the the ";" as %3B.  We've actually resorted to using "unicode" 
like representation (changing "U+03B" to U_03B so to not use a +) and 
transforming back when processing the incoming requests.

@Chuck, thanks for your snarky response. I'll leave it at that.

@Bill, thanks for mentioning the specific RFC referencing the encoding 
of ";".  That's what the team here figured out as well and it's nice to 
have an independent person verify our assumptions after reading the RFC 
sections.  Is it safe to assume that Tomcat code base assumes that 
anything after the ";" has to be the jsessionid?  That's our assumption 
at the moment.  Yes, there is a work around by using the 
request.getRequestURI()  instead as that has the whole URI and manually 
remove the "absolute" path to get the complete path info.

Since our framework is deployed on several different CFML servlets -- 
their implementation to get at the original http request wrapper differs 
a bit (three different vendors).  We'll probably stick to use the poor 
man's encoding using a modified unicode representation of ";" in the 
end.  Another solution is to write a filter and use the getRequestURI() 
and replace the bad path info in the request with the full length version.

Thanks again,
.Peter

Bill Barker said the following on 12/23/-28158 01:59 PM:
>> Shouldn't that be /index.cfm/somePathInfo&amp%3BwithMoreInfo/
>>
>> ?
>>
>> If you try the above URL, does it work?
>>
>> java.net.URLEncoder will encode ";" as "%3B".
>>
>> See the URL Specification (RFC 1738,
>> http://www.ietf.org/rfc/rfc1738.txt), section 2.2 "URL Character
>> Encoding Issues
>> ":
>>
>> "
>> Many URL schemes reserve certain characters for a special meaning:
>> their appearance in the scheme-specific part of the URL has a
>> designated semantics. If the character corresponding to an octet is
>> reserved in a scheme, the octet must be encoded.  The characters ";",
>> "/", "?", ":", "@", "=" and "&" are the characters which may be
>> reserved for special meaning within a scheme. No other characters may
>> be reserved within a scheme.
>> "
>>
>> The HTTP specification does not specifically say that semi-colons are
>> reserved, but perhaps the common interpretation of the URL spec is such
>> that semi-colons should always be encoded.
>>
>>     
>
> Actually it does, just by reference.  Section 3.2.1 of RFC 2616 defers to 
> RFC 2396.  And section 3.3 of that RFC gives a special meaning to ';'. 
> Tomcat doesn't handle this correctly according the the RFC, but no 
> developer/contributor has had enough of an itch to fix it.  But I doubt that 
> fixing it would help the OP much.
>
> The fully compliant Tomcat would have to remove anything after a ';' 
> (including the ';') up until the next '/' (if any) for the purpose of 
> mapping the request.  It should then re-include them in the various parts of 
> the request URI (except for ";jsessionid").  So it's a lot of work to 
> implement an archane feature that has plenty of work arounds.
>
>   
>> - -chris
>> -----BEGIN PGP SIGNATURE-----
>> Version: GnuPG v1.4.9 (MingW32)
>> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>>
>> iEYEARECAAYFAkpozVsACgkQ9CaO5/Lv0PBnLwCfXFSSIDAnRR0BurRKeS0ub/v9
>> 3UYAoJ1gp5oIqnJw2WgHx9LdVzqqAOAI
>> =rpT0
>> -----END PGP SIGNATURE----- 
>>     
>
>
>
>
>   

Re: request.getPathInfo() gets truncated when ";" is present

[Bill Barker <wb...@wilshire.com>]

"Christopher Schultz" <ch...@christopherschultz.net> wrote in message 
news:4A68CD5B.2090402@christopherschultz.net...
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Peter,
>
> On 7/23/2009 3:41 PM, Peter J. Farrell wrote:
>> /index.cfm/somePathInfo&amp;withMoreInfo/
>
> Shouldn't that be /index.cfm/somePathInfo&amp%3BwithMoreInfo/
>
> ?
>
> If you try the above URL, does it work?
>
> java.net.URLEncoder will encode ";" as "%3B".
>
> See the URL Specification (RFC 1738,
> http://www.ietf.org/rfc/rfc1738.txt), section 2.2 "URL Character
> Encoding Issues
> ":
>
> "
> Many URL schemes reserve certain characters for a special meaning:
> their appearance in the scheme-specific part of the URL has a
> designated semantics. If the character corresponding to an octet is
> reserved in a scheme, the octet must be encoded.  The characters ";",
> "/", "?", ":", "@", "=" and "&" are the characters which may be
> reserved for special meaning within a scheme. No other characters may
> be reserved within a scheme.
> "
>
> The HTTP specification does not specifically say that semi-colons are
> reserved, but perhaps the common interpretation of the URL spec is such
> that semi-colons should always be encoded.
>

Actually it does, just by reference.  Section 3.2.1 of RFC 2616 defers to 
RFC 2396.  And section 3.3 of that RFC gives a special meaning to ';'. 
Tomcat doesn't handle this correctly according the the RFC, but no 
developer/contributor has had enough of an itch to fix it.  But I doubt that 
fixing it would help the OP much.

The fully compliant Tomcat would have to remove anything after a ';' 
(including the ';') up until the next '/' (if any) for the purpose of 
mapping the request.  It should then re-include them in the various parts of 
the request URI (except for ";jsessionid").  So it's a lot of work to 
implement an archane feature that has plenty of work arounds.

> - -chris
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.9 (MingW32)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAkpozVsACgkQ9CaO5/Lv0PBnLwCfXFSSIDAnRR0BurRKeS0ub/v9
> 3UYAoJ1gp5oIqnJw2WgHx9LdVzqqAOAI
> =rpT0
> -----END PGP SIGNATURE----- 




---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: request.getPathInfo() gets truncated when ";" is present

[Christopher Schultz <ch...@christopherschultz.net>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Peter,

On 7/23/2009 3:41 PM, Peter J. Farrell wrote:
> /index.cfm/somePathInfo&withMoreInfo/

Shouldn't that be /index.cfm/somePathInfo&amp%3BwithMoreInfo/

?

If you try the above URL, does it work?

java.net.URLEncoder will encode ";" as "%3B".

See the URL Specification (RFC 1738,
http://www.ietf.org/rfc/rfc1738.txt), section 2.2 "URL Character
Encoding Issues
":

"
Many URL schemes reserve certain characters for a special meaning:
their appearance in the scheme-specific part of the URL has a
designated semantics. If the character corresponding to an octet is
reserved in a scheme, the octet must be encoded.  The characters ";",
"/", "?", ":", "@", "=" and "&" are the characters which may be
reserved for special meaning within a scheme. No other characters may
be reserved within a scheme.
"

The HTTP specification does not specifically say that semi-colons are
reserved, but perhaps the common interpretation of the URL spec is such
that semi-colons should always be encoded.

- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkpozVsACgkQ9CaO5/Lv0PBnLwCfXFSSIDAnRR0BurRKeS0ub/v9
3UYAoJ1gp5oIqnJw2WgHx9LdVzqqAOAI
=rpT0
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: request.getPathInfo() gets truncated when ";" is present

[André Warnier <aw...@ice-sa.com>]

Caldarale, Charles R wrote:
>> From: Peter J. Farrell [mailto:peter@mach-ii.com]
>> Subject: request.getPathInfo() gets truncated when ";" is present
>>
>> I'm trying to figure out a particular strange problem with
>> request.getPathInfo() data being truncated.
> 
> It's unfortunate that the mailing list doesn't have a filter that automatically bounces postings from people who can't be bothered to include the version of Tomcat they're using.
> 
Adding insult to injury, in a matter of speaking, is the fact that the 
shown encoding scheme for URI's is, in my view, totally wrong.
& -> &  is a correct encoding for HTML and XML, but not for URI's.

And URI's containing "&" and ";" characters in the path are, at the very 
least, "cruising for a bruising" ?

In other words, Peter, I have the definite impression that this which 
you call "strange problem with request.getPathInfo() data being 
truncated" is in fact not a problem of Tomcat at all, it is a problem 
with your application logic.

You should re-read the following carefully :
http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.2.2
and also section "2. URI Characters and Escape Sequences" of the RFC2396.

Both "&" and ";" are "reserved" characters, used as separators between 
elements.  For instance, ";" separates "parameters" from other path 
components.
I would not be surprised if you tried request.getParameters(), and you 
obtained some of the bits you say you're missing.




---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

RE: request.getPathInfo() gets truncated when ";" is present

["Caldarale, Charles R" <Ch...@unisys.com>]

> From: Peter J. Farrell [mailto:peter@mach-ii.com]
> Subject: request.getPathInfo() gets truncated when ";" is present
> 
> I'm trying to figure out a particular strange problem with
> request.getPathInfo() data being truncated.

It's unfortunate that the mailing list doesn't have a filter that automatically bounces postings from people who can't be bothered to include the version of Tomcat they're using.

 - Chuck


THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.