You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tomcat.apache.org by Mark Thomas <ma...@apache.org> on 2020/03/13 14:35:42 UTC
Proposed changes to UDecoder.ALLOW_ENCODED_SLASH
Hi all,
I am writing this up as this is a change I'd like to make in Tomcat 10
that I think is important to get right. It may also get back-ported.
This first arose in this mod_jk bug:
https://bz.apache.org/bugzilla/show_bug.cgi?id=62459
Ignoring the mod_jk aspects for now (they will come later) the bug
report raises the important question of how to handle the case where the
ID for a resource in a RESTful API includes a "/".
At the moment, Tomcat does not handle this correctly. If
ALLOW_ENCODED_SLASH is false, the request is rejected. If it is true,
the wrong resource identifier will be used. This is an edge case, but
one I'd like to fix.
My research led me back to RFC 3986. Quoting from section 2.2:
<quote>
The purpose of reserved characters is to provide a set of delimiting
characters that are distinguishable from other data within a URI.
URIs that differ in the replacement of a reserved character with its
corresponding percent-encoded octet are not equivalent. Percent-
encoding a reserved character, or decoding a percent-encoded octet
that corresponds to a reserved character, will change how the URI is
interpreted by most applications. Thus, characters in the reserved
set are protected from normalization and are therefore safe to be
used by scheme-specific and producer-specific algorithms for
delimiting data subcomponents within a URI.
</quote>
My reading of this is that there are some %nn sequences that we should
*never* decode. The values we pass to applications for ServletPath,
PathInfo etc. should still include these %nn sequences and the
application should decode them.
My next thought was "Which %nn sequences should be leave alone?". That
got me thinking about URIEncoding values and how to differentiate
between a %nn sequence we wanted to leave alone and the same sequence
appearing where a code point is represented by multiple bytes.
Fortunately, RFC7230 saves us from that complication as it requires all
encodings to be supersets of US-ASCII. Or to put is another way, the
only time %nn appears where nn is in the range 00 to 7F that %nn
sequence will *always* be representing the equivalent US-ASCII code point.
So, that simplifies things a little as we go back to considering which
%nn sequences we have to leave alone.
The starting point is "reserved" characters. From RFC 3986:
reserved = gen-delims / sub-delims
gen-delims = ":" / "/" / "?" / "#" / "[" / "]" / "@"
sub-delims = "!" / "$" / "&" / "'" / "(" / ")"
/ "*" / "+" / "," / ";" / "="
We are talking about URIs in Tomcat which, at the point we %nn decode,
is just the path. The path parameters and query string have been removed.
From RFC 7230:
absolute-path = 1*( "/" segment )
and from RFC 3986:
segment = *pchar
pchar = unreserved / pct-encoded / sub-delims / ":" / "@"
So the question is, which reserved characters cannot be safely decoded
from their %nn form.
We know all subdelims because:
- they are valid characters in a segment and with the query string and
path parameters removed, none of those characters have special meaning
That leaves gen-delims
Of those ":" and "@" are explicitly allowed in a segment. So that leaves:
"/" "?" "#" "[" "]"
"?" is the query delimiter but the query string has been removed so it
is safe to %nn decode to "?".
"#" is the fragment delimiter. The fragment will never reach the server
so it is safe to %nn decode to "#".
"[" and "]" are delimiters in the host but not the path so they are safe.
That just leaves "/".
My proposal is, therefore, actually very simple:
1. Remove the UDecoder.ALLOW_ENCODED_SLASH option.
2. Replace it with a per Connector setting that has three options:
a) deny (equivalent to ALLOW_ENCODED_SLASH="false")
b) decode (equivalent to ALLOW_ENCODED_SLASH="true")
c) allow (leaves as is)
Thoughts?
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: Proposed changes to UDecoder.ALLOW_ENCODED_SLASH
Posted by Mark Thomas <ma...@apache.org>.
On 14/03/2020 10:08, Oleg Kalnichevski wrote:
<snip/>
> I am not really qualified to comment on the proposal but as far as I
> can see it makes sense.
>
> Client side libraries however have an extra problem to contend with.
>
> Most of the time HttpClient just passes the request URI to server as
> is, exactly as specified by the user.
>
> There is a catch though. Absolute request URIs need to be parsed and
> split into respective authority and path/query components.
>
> http://host:8080/stuff/123
>
> ---
> GET /stuff/123 HTTP/1.1
> Host: host:8080
> ---
>
> However there is a fringe case that can cause creation of ambiguous or
> illegal request messages.
>
> What is one supposed to do with absolute request URIs like this one?
>
> http://host:8080//stuff///123
>
> There does not appear to be any statement in the RFC7230 as to what the
> expected behavior should be.
In discussion with the httpd folks it appears that the consensus view
there is that "/a//b/" != "/a/b/"
The definitions in RFC 3986 allow a URI of "/a//".
For two reasons Tomcat takes a different view:
1. Applications constructing URIs often get it wrong and add an extra
"/" where it is not necessary. If Tomcat didn't collapse these, those
applications would break.
2. When serving static resources, file systems will collapse "//" to
"/".
> What we presently do in both 4.x and 5.x release lines is normalizing
> such request URIs by collapsing multiple consecutive forward slashes in
> the path component into a single one.
Tomcat will do that as well for URIs it receives.
It could be argued that Tomcat should not do this. If we ever have a
user with a use case for not collapsing them then I suspect we'll add an
option to make that behaviour optional.
> By default HttpClient would generate this request as
> ---
> GET /stuff/123 HTTP/1.1
> Host: host:8080
> ---
>
> and not as
> ---
> GET //stuff///123 HTTP/1.1
> Host: host:8080
> ---
>
> That should not be relevant as far as this proposal is concerned but I
> thought I should mention it just in case as handling of consecutive
> forward slashes in the path component of request URIs had caused us a
> lot of grief in the past.
Thanks. It is useful to see where difficulties have occurred in the past.
Tomcat would treat the above 2 requests as the same but I understand
that httpd would not.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: Proposed changes to UDecoder.ALLOW_ENCODED_SLASH
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Fri, 2020-03-13 at 16:02 +0100, Michael Osipov wrote:
> Am 2020-03-13 um 15:35 schrieb Mark Thomas:
> > Hi all,
> >
> > I am writing this up as this is a change I'd like to make in Tomcat
> > 10
> > that I think is important to get right. It may also get back-
> > ported.
> >
> > This first arose in this mod_jk bug:
> > https://bz.apache.org/bugzilla/show_bug.cgi?id=62459
> >
> > Ignoring the mod_jk aspects for now (they will come later) the bug
> > report raises the important question of how to handle the case
> > where the
> > ID for a resource in a RESTful API includes a "/".
> >
> > At the moment, Tomcat does not handle this correctly. If
> > ALLOW_ENCODED_SLASH is false, the request is rejected. If it is
> > true,
> > the wrong resource identifier will be used. This is an edge case,
> > but
> > one I'd like to fix.
> >
> > My research led me back to RFC 3986. Quoting from section 2.2:
> >
> > <quote>
> > The purpose of reserved characters is to provide a set of
> > delimiting
> > characters that are distinguishable from other data within a URI.
> > URIs that differ in the replacement of a reserved character with
> > its
> > corresponding percent-encoded octet are not equivalent. Percent-
> > encoding a reserved character, or decoding a percent-encoded octet
> > that corresponds to a reserved character, will change how the URI
> > is
> > interpreted by most applications. Thus, characters in the reserved
> > set are protected from normalization and are therefore safe to be
> > used by scheme-specific and producer-specific algorithms for
> > delimiting data subcomponents within a URI.
> > </quote>
> >
> > My reading of this is that there are some %nn sequences that we
> > should
> > *never* decode. The values we pass to applications for ServletPath,
> > PathInfo etc. should still include these %nn sequences and the
> > application should decode them.
> >
> > My next thought was "Which %nn sequences should be leave alone?".
> > That
> > got me thinking about URIEncoding values and how to differentiate
> > between a %nn sequence we wanted to leave alone and the same
> > sequence
> > appearing where a code point is represented by multiple bytes.
> > Fortunately, RFC7230 saves us from that complication as it requires
> > all
> > encodings to be supersets of US-ASCII. Or to put is another way,
> > the
> > only time %nn appears where nn is in the range 00 to 7F that %nn
> > sequence will *always* be representing the equivalent US-ASCII code
> > point.
> >
> > So, that simplifies things a little as we go back to considering
> > which
> > %nn sequences we have to leave alone.
> >
> > The starting point is "reserved" characters. From RFC 3986:
> >
> > reserved = gen-delims / sub-delims
> >
> > gen-delims = ":" / "/" / "?" / "#" / "[" / "]" / "@"
> >
> > sub-delims = "!" / "$" / "&" / "'" / "(" / ")"
> > / "*" / "+" / "," / ";" / "="
> >
> > We are talking about URIs in Tomcat which, at the point we %nn
> > decode,
> > is just the path. The path parameters and query string have been
> > removed.
> >
> > > From RFC 7230:
> >
> > absolute-path = 1*( "/" segment )
> >
> > and from RFC 3986:
> >
> > segment = *pchar
> >
> > pchar = unreserved / pct-encoded / sub-delims / ":" / "@"
> >
> >
> > So the question is, which reserved characters cannot be safely
> > decoded
> > from their %nn form.
> >
> > We know all subdelims because:
> > - they are valid characters in a segment and with the query string
> > and
> > path parameters removed, none of those characters have special
> > meaning
> >
> > That leaves gen-delims
> >
> > Of those ":" and "@" are explicitly allowed in a segment. So that
> > leaves:
> >
> > "/" "?" "#" "[" "]"
> >
> > "?" is the query delimiter but the query string has been removed so
> > it
> > is safe to %nn decode to "?".
> >
> > "#" is the fragment delimiter. The fragment will never reach the
> > server
> > so it is safe to %nn decode to "#".
> >
> > "[" and "]" are delimiters in the host but not the path so they are
> > safe.
> >
> > That just leaves "/".
> >
> > My proposal is, therefore, actually very simple:
> >
> > 1. Remove the UDecoder.ALLOW_ENCODED_SLASH option.
> > 2. Replace it with a per Connector setting that has three options:
> > a) deny (equivalent to ALLOW_ENCODED_SLASH="false")
> > b) decode (equivalent to ALLOW_ENCODED_SLASH="true")
> > c) allow (leaves as is)
>
> I am CC'ing our expert olegk@ on this topic because at HttpComponents
> we
> had numerous JIRA issues regarding the handling and RFC 3986
> interpretation. It is, sadly, a constant source of trouble.
>
> Oleg, can you share your view on Mark's proposal?
>
> Michael
>
Michael et al
I am not really qualified to comment on the proposal but as far as I
can see it makes sense.
Client side libraries however have an extra problem to contend with.
Most of the time HttpClient just passes the request URI to server as
is, exactly as specified by the user.
There is a catch though. Absolute request URIs need to be parsed and
split into respective authority and path/query components.
http://host:8080/stuff/123
---
GET /stuff/123 HTTP/1.1
Host: host:8080
---
However there is a fringe case that can cause creation of ambiguous or
illegal request messages.
What is one supposed to do with absolute request URIs like this one?
http://host:8080//stuff///123
There does not appear to be any statement in the RFC7230 as to what the
expected behavior should be.
What we presently do in both 4.x and 5.x release lines is normalizing
such request URIs by collapsing multiple consecutive forward slashes in
the path component into a single one.
By default HttpClient would generate this request as
---
GET /stuff/123 HTTP/1.1
Host: host:8080
---
and not as
---
GET //stuff///123 HTTP/1.1
Host: host:8080
---
That should not be relevant as far as this proposal is concerned but I
thought I should mention it just in case as handling of consecutive
forward slashes in the path component of request URIs had caused us a
lot of grief in the past.
Oleg
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: Proposed changes to UDecoder.ALLOW_ENCODED_SLASH
Posted by Michael Osipov <mi...@apache.org>.
Am 2020-03-13 um 15:35 schrieb Mark Thomas:
> Hi all,
>
> I am writing this up as this is a change I'd like to make in Tomcat 10
> that I think is important to get right. It may also get back-ported.
>
> This first arose in this mod_jk bug:
> https://bz.apache.org/bugzilla/show_bug.cgi?id=62459
>
> Ignoring the mod_jk aspects for now (they will come later) the bug
> report raises the important question of how to handle the case where the
> ID for a resource in a RESTful API includes a "/".
>
> At the moment, Tomcat does not handle this correctly. If
> ALLOW_ENCODED_SLASH is false, the request is rejected. If it is true,
> the wrong resource identifier will be used. This is an edge case, but
> one I'd like to fix.
>
> My research led me back to RFC 3986. Quoting from section 2.2:
>
> <quote>
> The purpose of reserved characters is to provide a set of delimiting
> characters that are distinguishable from other data within a URI.
> URIs that differ in the replacement of a reserved character with its
> corresponding percent-encoded octet are not equivalent. Percent-
> encoding a reserved character, or decoding a percent-encoded octet
> that corresponds to a reserved character, will change how the URI is
> interpreted by most applications. Thus, characters in the reserved
> set are protected from normalization and are therefore safe to be
> used by scheme-specific and producer-specific algorithms for
> delimiting data subcomponents within a URI.
> </quote>
>
> My reading of this is that there are some %nn sequences that we should
> *never* decode. The values we pass to applications for ServletPath,
> PathInfo etc. should still include these %nn sequences and the
> application should decode them.
>
> My next thought was "Which %nn sequences should be leave alone?". That
> got me thinking about URIEncoding values and how to differentiate
> between a %nn sequence we wanted to leave alone and the same sequence
> appearing where a code point is represented by multiple bytes.
> Fortunately, RFC7230 saves us from that complication as it requires all
> encodings to be supersets of US-ASCII. Or to put is another way, the
> only time %nn appears where nn is in the range 00 to 7F that %nn
> sequence will *always* be representing the equivalent US-ASCII code point.
>
> So, that simplifies things a little as we go back to considering which
> %nn sequences we have to leave alone.
>
> The starting point is "reserved" characters. From RFC 3986:
>
> reserved = gen-delims / sub-delims
>
> gen-delims = ":" / "/" / "?" / "#" / "[" / "]" / "@"
>
> sub-delims = "!" / "$" / "&" / "'" / "(" / ")"
> / "*" / "+" / "," / ";" / "="
>
> We are talking about URIs in Tomcat which, at the point we %nn decode,
> is just the path. The path parameters and query string have been removed.
>
>>From RFC 7230:
>
> absolute-path = 1*( "/" segment )
>
> and from RFC 3986:
>
> segment = *pchar
>
> pchar = unreserved / pct-encoded / sub-delims / ":" / "@"
>
>
> So the question is, which reserved characters cannot be safely decoded
> from their %nn form.
>
> We know all subdelims because:
> - they are valid characters in a segment and with the query string and
> path parameters removed, none of those characters have special meaning
>
> That leaves gen-delims
>
> Of those ":" and "@" are explicitly allowed in a segment. So that leaves:
>
> "/" "?" "#" "[" "]"
>
> "?" is the query delimiter but the query string has been removed so it
> is safe to %nn decode to "?".
>
> "#" is the fragment delimiter. The fragment will never reach the server
> so it is safe to %nn decode to "#".
>
> "[" and "]" are delimiters in the host but not the path so they are safe.
>
> That just leaves "/".
>
> My proposal is, therefore, actually very simple:
>
> 1. Remove the UDecoder.ALLOW_ENCODED_SLASH option.
> 2. Replace it with a per Connector setting that has three options:
> a) deny (equivalent to ALLOW_ENCODED_SLASH="false")
> b) decode (equivalent to ALLOW_ENCODED_SLASH="true")
> c) allow (leaves as is)
I am CC'ing our expert olegk@ on this topic because at HttpComponents we
had numerous JIRA issues regarding the handling and RFC 3986
interpretation. It is, sadly, a constant source of trouble.
Oleg, can you share your view on Mark's proposal?
Michael
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org