[jira] Commented: (SLING-508) Parameter decoding uses wrong default charset

["Felix Meschberger (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SLING-508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12602634#action_12602634 ] 

Felix Meschberger commented on SLING-508:
-----------------------------------------

First off: Servlet Container parameters are not re-encoded by Sling (any more). They are taken as is.

Now, to what happens here:

On the one hand, the W3C [1] recomends browser vendors to encode non-ASCI characters in URLs in UTF-8. This should IMO also include the encoding of parameters in application/x-www-formurlencoded POSTed parameters, altough I could not find a real codification of this.

On the other hand, the Servlet Specification states, that all data read from POSTed content should be decoded with ISO-8859-1 encoding (Servlet API 2.4, Section 4.9). As servlet containers only read application/x-www-formurlencoded POST requests this issue is about these parameters.

Third, servlet containers are implemented inconsistently: Some (e.g. Tomcat) apply the Servlet API spec and read the data as ISO-8859-1 and some apply (e.g. Jetty) the W3C recommendation and read the data as UTF-8.

Fourth, browsers do not apply the W3C recomendation but instead encode the parameters in the character encoding of the page on which the form is placed.

Consider now the situation of a Servlet API conforming servlet container accepting form data of an UTF-8 encoded page: The parameters are encoded in UTF-8 and servlet container decodes this as ISO-8859-1 giving unreadable data. Conversely, if running in a W3C conforming container accepting form data of an ISO-8859-1 encoded page, the data will also be corrupt due to UTF-8 decoding of ISO-8859-1 data.

To come around this, we have very lilttle power. Best we can do is try to force the servlet container in decoding the parameter data in ISO-8859-1 and then to recode the raw data in whatever character encoding has been declared with the "_charset_" request parameter.

Two remarks:
(1) We use ISO-8859-1 because this encoding defines a 1:1 mapping of raw bytes to characters. In fact, the lower 256 characters of Unicode are exactly the characters from the ISO-8859-1 encoding. Thus ISO-8859-1 is kind of an identity encoding.
(2) "Trying to force" the container means, that we ensure the correct character set to use for reading the input, but if the input has already been read (e.g. by a filter outside Sling), we can not do much any more. This is probably not much of an issue, but we must be aware of it.


[1] http://www.w3.org/TR/html4/appendix/notes.html#h-B.2.1

> Parameter decoding uses wrong default charset
> ---------------------------------------------
>
>                 Key: SLING-508
>                 URL: https://issues.apache.org/jira/browse/SLING-508
>             Project: Sling
>          Issue Type: Bug
>          Components: Engine
>    Affects Versions: 2.0.0
>            Reporter: Tobias Bocanegra
>            Assignee: Felix Meschberger
>            Priority: Blocker
>
> As of SLING-152 the request paremeters are re-encoded if a _charset_ parameter is present. it assumes that the default encoding is
> UTF-8 which is not the case for servlet spec compliant containers (eg. tomcat).
> change the default encoding to ISO-8851-1 or make it configurable.
> see: http://svn.apache.org/viewvc/incubator/sling/trunk/engine/src/main/java/org/apache/sling/engine/impl/parameters/Util.java?view=markup

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.