You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Christoph Pirkl <cp...@hotmail.com> on 2008/06/23 12:38:48 UTC

UTF-8 handling differs between two servlets within the same application

Hi,
 
we are developing an application, where we experience an interesting UTF-8 related behaviour.
 
For the development the team mostly uses Tomcat, since the deployment is much faster and nicer. All tests are repeated on WebSphere, because that is what our customer runs.
 
We are running a Tomcat environment
 
Tomcat 6.0.16
Java 1.6 - 6
Struts 1.2.9
RHEL 4 and Windows XP / SP2
 
and a WebSphere environment
 
WebSphere 6.0.2.17
Java 1.4 (shipped with WS)
Struts 1.2.9
RHEL 4
 
There is a login action and an action where the user can change his password.
 
Both forms use the "post"-method. Content of both HTML-pages is declared as UTF-8. We use an encoding filter (Tomcat only) as described in:
 
http://wiki.apache.org/tomcat/Tomcat/UTF-8
 
Funny thing is:
 
One of the servlets (login) works as expected.
Username and password are in proper UTF-8 encoding.
 
The other servlet does not receive proper UTF-8.
 
This way the password-hashes are corrupted if the user types in non-ANSI chars (i.e. every char that needs multibyte encoding in UTF-8). The user then cannot log into the application.
 
In both cases the filter code is executed, we verified that with a debugger.
 
Tracing the client-server communication with WireShark shows that the Web-Browser sends identical password data as a result of both forms.
 
In the request object on the other hand the strings differ if a special character was entered. We checked that with a debugger by setting a breakpoint in the encoding filter.
 
We added URIEncoding="UTF-8" in the connector, as suggested in various posts we found googling the problem... unfortunately to no avail.
 
If I'm not totally mistaken there is no application code run before the filter, so either we are making a mistake that has slipped us (quite possible) or we might have found a rare bug in Tomcat.
 
On WebSphere the same code works without a problem.
 
I would appreciate any kind of hint, since developing on Tomcat is much more fun than on WebSphere.
 
Thanks for your help in advance and with kind regards,
 
Christoph.
_________________________________________________________________
Die aktuelle Frühjahrsmode - Preise vergleichen bei MSN Shopping
http://shopping.msn.de/category/damenbekleidung/bcatid66/forsale?text=category:damenbekleidung&edt=1&ptnrid=230
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Re: UTF-8 handling differs between two servlets within the same application

Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

All,

Christopher Schultz wrote:
| The filter example on this page needs to be
| fixed so that the encoding is only set if one is not detected. This is a
| BIG BUG in the filter shown on that page, and someone should fix it
| (maybe I will... I just registered for the Wiki).

Okay, I have fixed the filter to it does not clobber client-specified
character encoding.

- -chris

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkhf64sACgkQ9CaO5/Lv0PAXjQCeJ79ftq4Ai5q2QTtnwBkifLtp
weoAnjeNcpDtzExJvCSQWGvwOGE98lEM
=SAzM
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Re: UTF-8 handling differs between two servlets within the same application

Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

André,

André Warnier wrote:
| I am sorry to butt in again, but are you *really* sure that the problem
| is not earlier in the chain than what you think ?
| I have read the article at the link given earlier :
| http://wiki.apache.org/tomcat/Tomcat/UTF-8
| and I am quite sure that what is said in that article is wrong, or at
| least incomplete.  The article seems to assume that whatever the browser
| sends is always iso-8859-1, and that at the server level you can then
| just go and "decode" it into utf-8.  That is wrong, I can assure you.

You're right: you can't just assume that the incoming data is UTF-8. The
problem is that browsers often do not send a Content-Type encoding
string along with all POST requests. They /should/, but sometimes they
do not. In these cases, the server is left to guess. Guessing is hard,
but most browsers act somewhat predictably...

| Browsers will send utf-8 if the right conditions are met, and you will
| corrupt that data if you force it through a second encoding/decoding.
| Browsers will also sometimes send iso-8859-1, if you are not careful or
| if the browser is buggy. It happens.  (iso-8859-1 is the default in
| HTTP, so if you do not specify things differently, that is what you'll
get).

Most browsers will send request #1 in the same encoding that was used
for response #0. That is, if a page is encoded in UTF8, then the
encoding using to submit from that page (unless otherwise specified)
will use the same encoding -- even if that encoding is not specified in
the Content-Type header.

| In an ideal world, when a browser sends a string parameter via a POST,
| each parameter value should be enclosed in a part with a header and a
| content. The header of the part should have a line
| Content-type: text/plain; charset=xxxxx
| and the content of that part should then be in that xxxx charset encoding.

"parting" is not required, here. You just encode the whole POST with the
same encoding, and use the standard Content-Type header including the
encoding.

Now, back to the server. No server should ever clobber an encoding
specified by the client. The filter example on this page needs to be
fixed so that the encoding is only set if one is not detected. This is a
BIG BUG in the filter shown on that page, and someone should fix it
(maybe I will... I just registered for the Wiki).

If you /know/ that your pages are being sent in UTF-8 and you make a
reasonable assumption that requests with no Content-Type encoding will
use the encoding of the previous response, then the filter listed on the
aforementioned page is acceptable (again, with a check for an existing
content type encoding).

| It is quite possible that Tomcat's innards do not do things correctly
| when they decode a POST, and just deliver the raw parameter value as
| received.  But that would surprise me, and I would submit that it would
| then be a bug.

Tomcat does, in fact, decode the parameters properly. That's what the
setCharacterEncoding parameter does -- it sets the character encoding
that will be used by any Reader used to read the request's body. Your
code does not have to do anything special.

- -chris

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkhf4xkACgkQ9CaO5/Lv0PBWuACePccDgzP9kudNTq6v7d88qe98
KowAoILM6V+uJESshpiSQOGfAnvdDGA1
=4a8J
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


RE: UTF-8 handling differs between two servlets within the same application

Posted by Christoph Pirkl <cp...@hotmail.com>.
Hi all,

thank you very much for your helpful answers!

We could resolve the encoding problem by adding the attribute

enctype="multipart/form-data"

to the html form. All special chars are received correctly now. Additionally we will add accept-charset="UTF-8" to avoid further problems.

Kind regards,

Christoph.
_________________________________________________________________
Lustige Emoticons für Ihren Messenger! Hier kostenlos downloaden!
http://messenger.live.de/mein/
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Re: UTF-8 handling differs between two servlets within the same application

Posted by Mark Thomas <ma...@apache.org>.
André Warnier wrote:
> 
> 
> Mark Thomas wrote:
>>
>> I tend to use the following as a starting point to check my config is 
>> OK. It is also useful to compare headers etc for your application 
>> against the headers from this simple test case.
>>
>> http://wiki.apache.org/tomcat/FAQ/CharacterEncoding#Q4
>>
> This is a bit outside the scope of this thread, but as someone 
> confronted with this kind of character sets issues in the web all the 
> time, I feel I have to say that the comment at the beginning of that 
> example can be misleading, and in my view should be taken out.

I think it would be better to leave it in (some people are stuck supporting 
apps like this) but add a warning along the lines of GET + forms == bad idea.

It is a wiki so you can edit it yourself ;)

Mark



---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Re: UTF-8 handling differs between two servlets within the same application

Posted by André Warnier <aw...@ice-sa.com>.

Mark Thomas wrote:
> 
> I tend to use the following as a starting point to check my config is 
> OK. It is also useful to compare headers etc for your application 
> against the headers from this simple test case.
> 
> http://wiki.apache.org/tomcat/FAQ/CharacterEncoding#Q4
> 
This is a bit outside the scope of this thread, but as someone 
confronted with this kind of character sets issues in the web all the 
time, I feel I have to say that the comment at the beginning of that 
example can be misleading, and in my view should be taken out.

It is of a nature to induce people into doing things they should not, 
and which would always bite them back in the end.
(For the same reason, I believe that all the methods or parameters 
dealing with "URI encoding" should be banned).

I can make a long case, but the summary is : don't use GET with forms, 
if you want to have any luck with applications that may have to handle 
input characters other than US-ASCII (as all web applications will have 
to, sooner or later; think of smileys).
The situation is already confusing enough with POSTed forms, without 
adding extra problem sources.

The HTML 4.01 spec (and, I suspect, the XHTML also) mentions this as 
follows, in the same RFC, same section :

Note. The "get" method restricts form data set values to ASCII 
characters. Only the "post" method (with enctype="multipart/form-data") 
is specified to cover the entire [ISO10646] character set.

(http://www.w3.org/TR/html401/interact/forms.html#submit-format
17.13.4 Form content types )
Also see RFC3986.

André

---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Re: server resources

Posted by Martin <mg...@hotmail.com>.
assuming you'll want to manage multiple resources such as DB 
connection(pools) and/or webservices and/or JMS Messaging Queues
havent heard about <Apache> PHP's ability to multi-thread?

anyone?
Martin
----- Original Message ----- 
From: "Johnny Kewl" <jo...@kewlstuff.co.za>
To: "Tomcat Users List" <us...@tomcat.apache.org>
Sent: Monday, June 23, 2008 7:28 PM
Subject: Re: server resources


>
> ----- Original Message ----- 
> From: "Marcos Molina" <mm...@adinet.com.uy>
> To: "Tomcat Users List" <us...@tomcat.apache.org>
> Sent: Monday, June 23, 2008 11:49 PM
> Subject: server resources
>
>
>> Hi people!
>>
>> I´m working with tomcat for years, in intranet envoiroment.
>> And now i need to put some works in anothers servers, and then i talk 
>> about tomcat with another people, but most servers admins tell me that 
>> TomCat need too much resoruces from machines. Is that right ??
>> Thanks for your response.
>>
>> Marcos
>
> Marcos, you can find the answer for yourself... google for "Java vs PHP"
> You'll find a healthy dose of pro, cons, bias, prejudice, hatred, 
> groupies, and marketing?
>
> Java is heavier on a machine... its got to load up the JVM.
> ... that initial hit is probably in the order of 200 megs versus 50... 
> then its what the app does.
>
> PHP is more widely used, its easier for easy stuff.
> But if a SP tells me that I got to learn another language, vs buying 
> another 500 m chip, or hosting our box... I'd just change my SP ;) 
> Retraining staff costs a hell of a lot more.
> Find a SP that specializes in hosting Tomcat... from the sounds of things 
> your SP doesnt know Java.
>
> Actually TC should do that... list all TC friendly SP's.
>
> Then if you talking to a .Net SP... Tomcats really going to suck... its 
> free ;)
>
> Google, for the swings and balances... the answer depends very much on the 
> fan club you talking to ;)
>
> ---------------------------------------------------------------------------
> HARBOR : http://www.kewlstuff.co.za/index.htm
> The most powerful application server on earth.
> The only real POJO Application Server.
> See it in Action : http://www.kewlstuff.co.za/cd_tut_swf/whatisejb1.htm
> ---------------------------------------------------------------------------
>
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
> 


---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Re: server resources

Posted by Johnny Kewl <jo...@kewlstuff.co.za>.
----- Original Message ----- 
From: "Marcos Molina" <mm...@adinet.com.uy>
To: "Tomcat Users List" <us...@tomcat.apache.org>
Sent: Monday, June 23, 2008 11:49 PM
Subject: server resources


> Hi people!
>
> I´m working with tomcat for years, in intranet envoiroment.
> And now i need to put some works in anothers servers, and then i talk 
> about tomcat with another people, but most servers admins tell me that 
> TomCat need too much resoruces from machines. Is that right ??
> Thanks for your response.
>
> Marcos

Marcos, you can find the answer for yourself... google for "Java vs PHP"
You'll find a healthy dose of pro, cons, bias, prejudice, hatred, groupies, 
and marketing?

Java is heavier on a machine... its got to load up the JVM.
... that initial hit is probably in the order of 200 megs versus 50... then 
its what the app does.

PHP is more widely used, its easier for easy stuff.
But if a SP tells me that I got to learn another language, vs buying another 
500 m chip, or hosting our box... I'd just change my SP ;) Retraining staff 
costs a hell of a lot more.
Find a SP that specializes in hosting Tomcat... from the sounds of things 
your SP doesnt know Java.

Actually TC should do that... list all TC friendly SP's.

Then if you talking to a .Net SP... Tomcats really going to suck... its free 
;)

Google, for the swings and balances... the answer depends very much on the 
fan club you talking to ;)

---------------------------------------------------------------------------
HARBOR : http://www.kewlstuff.co.za/index.htm
The most powerful application server on earth.
The only real POJO Application Server.
See it in Action : http://www.kewlstuff.co.za/cd_tut_swf/whatisejb1.htm
--------------------------------------------------------------------------- 


---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


server resources

Posted by Marcos Molina <mm...@adinet.com.uy>.
Hi people!

I´m working with tomcat for years, in intranet envoiroment.
And now i need to put some works in anothers servers, and then i talk 
about tomcat with another people, but most servers admins tell me that 
TomCat need too much resoruces from machines. Is that right ??
Thanks for your response.

Marcos

---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Re: UTF-8 handling differs between two servlets within the same application

Posted by Mark Thomas <ma...@apache.org>.
André Warnier wrote:
> If you agree, a question to start in that process :
> In the login page, the <form> element should have the following 
> attributes :
> <form method="POST" enctype="multipart/form-data" accept-charset="UTF-8" 
> ...>
> Does it ?
> 
> Ref : http://www.w3.org/TR/html401/interact/forms.html#submit-format
> and in particular : 17.13.4 Form content types
> (among other things, the spec says that this is the only valid way to 
> submit non-ascii data - such as passwords containing "accented" characters)

I tend to use the following as a starting point to check my config is OK. 
It is also useful to compare headers etc for your application against the 
headers from this simple test case.

http://wiki.apache.org/tomcat/FAQ/CharacterEncoding#Q4

Mark



---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Re: UTF-8 handling differs between two servlets within the same application

Posted by André Warnier <aw...@ice-sa.com>.
Mark Thomas wrote:
> André Warnier wrote:
>> It is quite possible that Tomcat's innards do not do things correctly 
>> when they decode a POST, and just deliver the raw parameter value as 
>> received.  But that would surprise me, and I would submit that it 
>> would then be a bug.
> 
> As far as I am aware, Tomcat correctly decodes parameters for all 
> versions in all cases (provided the client sends sensible input). If you 
> have a test case that doesn't work, that would be a bug and should be 
> added to Bugzilla.
> 
> Mark
> 

To the Original Poster :

1) We have it thus by a Tomcat authority (above) that Tomcat correctly 
decodes the POST parameters, IF they are submitted correctly by the 
browser.  That means that it should not be necessary to add a filter, 
nor to literally specify a request character set, IF the form parameters 
are being submitted correctly by the browser.

2) You are seeing a different thing in your application, where 
parameters (in this case the userid or password from the form), do NOT 
come in as you expect (iow proper Unicode reflecting what the user typed 
in the form).

3) There are only two possible conclusions :
a) the parameters from the form are not being submitted properly by the 
browser, whatever the reason is.
b) the parameters are being submitted properly by the browser, but there 
is a bug in Tomcat.

I would imagine that if Tomcat had a bug in that respect, it would 
affect a lot of applications, and would have been all over this forum 
and others many times already.  That does not seem to be the case.

So I would suggest that what you would need to investigate first is : 
are the parameters *really* coming in properly from the browser ?
If not, there are many possible reasons, which can be checked out one by 
one and eliminated.  In the process, it will also make the application 
more robust.  And it would also make your life easier if you could avoid 
a filter, and if the same application would work properly under both 
Tomcat and Websphere.

If you agree, a question to start in that process :
In the login page, the <form> element should have the following attributes :
<form method="POST" enctype="multipart/form-data" accept-charset="UTF-8" 
...>
Does it ?

Ref : http://www.w3.org/TR/html401/interact/forms.html#submit-format
and in particular : 17.13.4 Form content types
(among other things, the spec says that this is the only valid way to 
submit non-ascii data - such as passwords containing "accented" characters)

André




---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Re: UTF-8 handling differs between two servlets within the same application

Posted by Mark Thomas <ma...@apache.org>.
André Warnier wrote:
> It is quite possible that Tomcat's innards do not do things correctly 
> when they decode a POST, and just deliver the raw parameter value as 
> received.  But that would surprise me, and I would submit that it would 
> then be a bug.

As far as I am aware, Tomcat correctly decodes parameters for all versions 
in all cases (provided the client sends sensible input). If you have a test 
case that doesn't work, that would be a bug and should be added to Bugzilla.

Mark



---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Re: UTF-8 handling differs between two servlets within the same application

Posted by André Warnier <aw...@ice-sa.com>.
Christopher Schultz wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Youssef,
> 
> Youssef Mohammed wrote:

Guys,
I am sorry to butt in again, but are you *really* sure that the problem 
is not earlier in the chain than what you think ?
I have read the article at the link given earlier :
http://wiki.apache.org/tomcat/Tomcat/UTF-8
and I am quite sure that what is said in that article is wrong, or at 
least incomplete.  The article seems to assume that whatever the browser 
sends is always iso-8859-1, and that at the server level you can then 
just go and "decode" it into utf-8.  That is wrong, I can assure you. 
Browsers will send utf-8 if the right conditions are met, and you will 
corrupt that data if you force it through a second encoding/decoding.
Browsers will also sometimes send iso-8859-1, if you are not careful or 
if the browser is buggy. It happens.  (iso-8859-1 is the default in 
HTTP, so if you do not specify things diferent, that is what you'll get).

In an ideal world, when a browser sends a string parameter via a POST, 
each parameter value should be enclosed in a part with a header and a 
content. The header of the part should have a line
Content-type: text/plain; charset=xxxxx
and the content of that part should then be in that xxxx charset encoding.

The receiving server should decode each part of the POST, and if it does 
it's job right, should look at the Content-type header, and use it to 
decode the corresponding parameter into Unicode (if it isn't yet so), 
because that is what the request.getParameter() would expect to receive, 
since Java's internal charset is Unicode.

I know you may have examined the value sent, using some snooping 
software. But even if the value is the same in terms of bytes, but the 
Content-type header is different, the final result may not be what you 
expect.

It is quite possible that Tomcat's innards do not do things correctly 
when they decode a POST, and just deliver the raw parameter value as 
received.  But that would surprise me, and I would submit that it would 
then be a bug.

André


---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Re: UTF-8 handling differs between two servlets within the same application

Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Youssef,

Youssef Mohammed wrote:
| I don't think you need to encode again in the reset or validation. I
am not
| quite sure if calling setCharacterEncoding twice would be the problem
here.
| but just try to remove the redundant ones.

At best, calling setCharacterEncoding after parameters have been read
does nothing. At worst, it will throw an IllegalStateException (which
would be better, IMO).

|> Is it possible, that this character encoding filter is called too late,
|> i.e. when the submitted request parameters are already processed?
|
| Not sure, but if you have other filters they might process the request
| before your encoding filter.

Absolutely. If another filter before your utf8 filter calls anything
like request.getParameter, then the parameters will be read using the
current encoding.

You should make sure that your filter is the first one to run in the chain.

- -chris

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkhfwx8ACgkQ9CaO5/Lv0PCTHwCffZtngxoYN+xlPDJEos/eNBYc
AJYAnAj8q8Qs/t9TNsBjxyc09zIsNfZx
=Qe0B
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Re: UTF-8 handling differs between two servlets within the same application

Posted by Youssef Mohammed <yo...@gmail.com>.
On Mon, Jun 23, 2008 at 4:29 PM, Christoph Pirkl <cp...@hotmail.com> wrote:

>
> Hi Youssef,
>
> thank you for your tip, but we are already using a filter that does exactly
> the same thing. We also set the character encoding in the reset and
> validation methods of all action forms.


I don't think you need to encode again in the reset or validation. I am not
quite sure if calling setCharacterEncoding twice would be the problem here.
but just try to remove the redundant ones.


>
> Is it possible, that this character encoding filter is called too late,
> i.e. when the submitted request parameters are already processed?


Not sure, but if you have other filters they might process the request
before your encoding filter.


>
> Kind regards,
>
> Christoph.
>
> > Date: Mon, 23 Jun 2008 15:54:36 +0300
> > From: youssef.mohammed@gmail.com
> > To: users@tomcat.apache.org
> > Subject: Re: UTF-8 handling differs between two servlets within the same
> application
> >
> > Christoph;
> > We used to have the same issue two years ago with older version of
> > tomcat 4.x . And yes it was working just nice with Websphere.
> > We resolved that but just adding a filter that would always set the
> encoding
> > to utf8.
> >
> > namely ...
> > in web.xml
> >
> >
> > CharEncoding
> > com.company.ipo.utils.CharSetFilter
> >
> >
> >
> > CharEncoding
> > action
> >
> >
> >
> > and the class is ...
> >
> > public class CharSetFilter implements Filter {
> >
> > public void destroy() {
> > }
> >
> > public void doFilter(ServletRequest request , ServletResponse response,
> > FilterChain chain) throws IOException, ServletException {
> >
> > request.setCharacterEncoding("UTF-8");
> > chain.doFilter(request,response);
> > }
> >
> > public void init(FilterConfig arg0) throws ServletException {
> >
> > }
> >
> > }
> >
> >
> > On Mon, Jun 23, 2008 at 1:38 PM, Christoph Pirkl  wrote:
> >
> >>
> >> Hi,
> >>
> >> we are developing an application, where we experience an interesting
> UTF-8
> >> related behaviour.
> >>
> >> For the development the team mostly uses Tomcat, since the deployment is
> >> much faster and nicer. All tests are repeated on WebSphere, because that
> is
> >> what our customer runs.
> >>
> >> We are running a Tomcat environment
> >>
> >> Tomcat 6.0.16
> >> Java 1.6 - 6
> >> Struts 1.2.9
> >> RHEL 4 and Windows XP / SP2
> >>
> >> and a WebSphere environment
> >>
> >> WebSphere 6.0.2.17
> >> Java 1.4 (shipped with WS)
> >> Struts 1.2.9
> >> RHEL 4
> >>
> >> There is a login action and an action where the user can change his
> >> password.
> >>
> >> Both forms use the "post"-method. Content of both HTML-pages is declared
> as
> >> UTF-8. We use an encoding filter (Tomcat only) as described in:
> >>
> >> http://wiki.apache.org/tomcat/Tomcat/UTF-8
> >>
> >> Funny thing is:
> >>
> >> One of the servlets (login) works as expected.
> >> Username and password are in proper UTF-8 encoding.
> >>
> >> The other servlet does not receive proper UTF-8.
> >>
> >> This way the password-hashes are corrupted if the user types in non-ANSI
> >> chars (i.e. every char that needs multibyte encoding in UTF-8). The user
> >> then cannot log into the application.
> >>
> >> In both cases the filter code is executed, we verified that with a
> >> debugger.
> >>
> >> Tracing the client-server communication with WireShark shows that the
> >> Web-Browser sends identical password data as a result of both forms.
> >>
> >> In the request object on the other hand the strings differ if a special
> >> character was entered. We checked that with a debugger by setting a
> >> breakpoint in the encoding filter.
> >>
> >> We added URIEncoding="UTF-8" in the connector, as suggested in various
> >> posts we found googling the problem... unfortunately to no avail.
> >>
> >> If I'm not totally mistaken there is no application code run before the
> >> filter, so either we are making a mistake that has slipped us (quite
> >> possible) or we might have found a rare bug in Tomcat.
> >>
> >> On WebSphere the same code works without a problem.
> >>
> >> I would appreciate any kind of hint, since developing on Tomcat is much
> >> more fun than on WebSphere.
> >>
> >> Thanks for your help in advance and with kind regards,
> >>
> >> Christoph.
> >> ---------------------------------------------------------------------
> >> To start a new topic, e-mail: users@tomcat.apache.org
> >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >> For additional commands, e-mail: users-help@tomcat.apache.org
> >>
> >>
> >
> >
> > --
> > Regards, Youssef
>
> _________________________________________________________________
> Die aktuelle Frühjahrsmode - Preise vergleichen bei MSN Shopping
>
> http://shopping.msn.de/category/damenbekleidung/bcatid66/forsale?text=category:damenbekleidung&edt=1&ptnrid=230
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>


-- 
Regards, Youssef

RE: UTF-8 handling differs between two servlets within the same application

Posted by Christoph Pirkl <cp...@hotmail.com>.
Hi Youssef,

thank you for your tip, but we are already using a filter that does exactly the same thing. We also set the character encoding in the reset and validation methods of all action forms.

Is it possible, that this character encoding filter is called too late, i.e. when the submitted request parameters are already processed?

Kind regards,

Christoph.

> Date: Mon, 23 Jun 2008 15:54:36 +0300
> From: youssef.mohammed@gmail.com
> To: users@tomcat.apache.org
> Subject: Re: UTF-8 handling differs between two servlets within the same application
>
> Christoph;
> We used to have the same issue two years ago with older version of
> tomcat 4.x . And yes it was working just nice with Websphere.
> We resolved that but just adding a filter that would always set the encoding
> to utf8.
>
> namely ...
> in web.xml
>
> 
> CharEncoding
> com.company.ipo.utils.CharSetFilter
> 
>
> 
> CharEncoding
> action
> 
>
>
> and the class is ...
>
> public class CharSetFilter implements Filter {
>
> public void destroy() {
> }
>
> public void doFilter(ServletRequest request , ServletResponse response,
> FilterChain chain) throws IOException, ServletException {
>
> request.setCharacterEncoding("UTF-8");
> chain.doFilter(request,response);
> }
>
> public void init(FilterConfig arg0) throws ServletException {
>
> }
>
> }
>
>
> On Mon, Jun 23, 2008 at 1:38 PM, Christoph Pirkl  wrote:
>
>>
>> Hi,
>>
>> we are developing an application, where we experience an interesting UTF-8
>> related behaviour.
>>
>> For the development the team mostly uses Tomcat, since the deployment is
>> much faster and nicer. All tests are repeated on WebSphere, because that is
>> what our customer runs.
>>
>> We are running a Tomcat environment
>>
>> Tomcat 6.0.16
>> Java 1.6 - 6
>> Struts 1.2.9
>> RHEL 4 and Windows XP / SP2
>>
>> and a WebSphere environment
>>
>> WebSphere 6.0.2.17
>> Java 1.4 (shipped with WS)
>> Struts 1.2.9
>> RHEL 4
>>
>> There is a login action and an action where the user can change his
>> password.
>>
>> Both forms use the "post"-method. Content of both HTML-pages is declared as
>> UTF-8. We use an encoding filter (Tomcat only) as described in:
>>
>> http://wiki.apache.org/tomcat/Tomcat/UTF-8
>>
>> Funny thing is:
>>
>> One of the servlets (login) works as expected.
>> Username and password are in proper UTF-8 encoding.
>>
>> The other servlet does not receive proper UTF-8.
>>
>> This way the password-hashes are corrupted if the user types in non-ANSI
>> chars (i.e. every char that needs multibyte encoding in UTF-8). The user
>> then cannot log into the application.
>>
>> In both cases the filter code is executed, we verified that with a
>> debugger.
>>
>> Tracing the client-server communication with WireShark shows that the
>> Web-Browser sends identical password data as a result of both forms.
>>
>> In the request object on the other hand the strings differ if a special
>> character was entered. We checked that with a debugger by setting a
>> breakpoint in the encoding filter.
>>
>> We added URIEncoding="UTF-8" in the connector, as suggested in various
>> posts we found googling the problem... unfortunately to no avail.
>>
>> If I'm not totally mistaken there is no application code run before the
>> filter, so either we are making a mistake that has slipped us (quite
>> possible) or we might have found a rare bug in Tomcat.
>>
>> On WebSphere the same code works without a problem.
>>
>> I would appreciate any kind of hint, since developing on Tomcat is much
>> more fun than on WebSphere.
>>
>> Thanks for your help in advance and with kind regards,
>>
>> Christoph.
>> ---------------------------------------------------------------------
>> To start a new topic, e-mail: users@tomcat.apache.org
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>
>
> --
> Regards, Youssef

_________________________________________________________________
Die aktuelle Frühjahrsmode - Preise vergleichen bei MSN Shopping
http://shopping.msn.de/category/damenbekleidung/bcatid66/forsale?text=category:damenbekleidung&edt=1&ptnrid=230
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Re: UTF-8 handling differs between two servlets within the same application

Posted by Youssef Mohammed <yo...@gmail.com>.
Christoph;
    We used to have the same issue two years ago with older version of
tomcat 4.x . And yes it was working just nice with Websphere.
We resolved that but just adding a filter that would always set the encoding
to utf8.

namely ...
in web.xml

  <filter>
    <filter-name>CharEncoding</filter-name>
    <filter-class>com.company.ipo.utils.CharSetFilter</filter-class>
  </filter>

 <filter-mapping>
    <filter-name>CharEncoding</filter-name>
     <servlet-name>action</servlet-name>
 </filter-mapping>


and the class is ...

public class CharSetFilter implements Filter {

    public void destroy() {
    }

    public void doFilter(ServletRequest request , ServletResponse response,
            FilterChain chain) throws IOException, ServletException {

        request.setCharacterEncoding("UTF-8");
        chain.doFilter(request,response);
    }

    public void init(FilterConfig arg0) throws ServletException {

    }

}


On Mon, Jun 23, 2008 at 1:38 PM, Christoph Pirkl <cp...@hotmail.com> wrote:

>
> Hi,
>
> we are developing an application, where we experience an interesting UTF-8
> related behaviour.
>
> For the development the team mostly uses Tomcat, since the deployment is
> much faster and nicer. All tests are repeated on WebSphere, because that is
> what our customer runs.
>
> We are running a Tomcat environment
>
> Tomcat 6.0.16
> Java 1.6 - 6
> Struts 1.2.9
> RHEL 4 and Windows XP / SP2
>
> and a WebSphere environment
>
> WebSphere 6.0.2.17
> Java 1.4 (shipped with WS)
> Struts 1.2.9
> RHEL 4
>
> There is a login action and an action where the user can change his
> password.
>
> Both forms use the "post"-method. Content of both HTML-pages is declared as
> UTF-8. We use an encoding filter (Tomcat only) as described in:
>
> http://wiki.apache.org/tomcat/Tomcat/UTF-8
>
> Funny thing is:
>
> One of the servlets (login) works as expected.
> Username and password are in proper UTF-8 encoding.
>
> The other servlet does not receive proper UTF-8.
>
> This way the password-hashes are corrupted if the user types in non-ANSI
> chars (i.e. every char that needs multibyte encoding in UTF-8). The user
> then cannot log into the application.
>
> In both cases the filter code is executed, we verified that with a
> debugger.
>
> Tracing the client-server communication with WireShark shows that the
> Web-Browser sends identical password data as a result of both forms.
>
> In the request object on the other hand the strings differ if a special
> character was entered. We checked that with a debugger by setting a
> breakpoint in the encoding filter.
>
> We added URIEncoding="UTF-8" in the connector, as suggested in various
> posts we found googling the problem... unfortunately to no avail.
>
> If I'm not totally mistaken there is no application code run before the
> filter, so either we are making a mistake that has slipped us (quite
> possible) or we might have found a rare bug in Tomcat.
>
> On WebSphere the same code works without a problem.
>
> I would appreciate any kind of hint, since developing on Tomcat is much
> more fun than on WebSphere.
>
> Thanks for your help in advance and with kind regards,
>
> Christoph.
> _________________________________________________________________
> Die aktuelle Frühjahrsmode - Preise vergleichen bei MSN Shopping
>
> http://shopping.msn.de/category/damenbekleidung/bcatid66/forsale?text=category:damenbekleidung&edt=1&ptnrid=230
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>


-- 
Regards, Youssef

Re: UTF-8 handling differs between two servlets within the same application

Posted by André Warnier <aw...@ice-sa.com>.
Hi.

Christoph Pirkl wrote:
[...]

Not being a Tomcat/Websphere expert myself, but having had to track down 
similar problems before, just a couple of notes :

If this is true :

> Both forms use the "post"-method. 

then the userid and password are sent in the body of the request, not in 
the URI.

and thus this

 > We added URIEncoding="UTF-8" in the connector, as suggested in 
various posts we found googling the problem... unfortunately to no avail.

.. should not have any effect, because if the name matches the function, 
it would affect only the URI, not the body.

On the other hand :

> Content of both HTML-pages is declared as UTF-8. 

How exactly ?
There are 3 elements that can play a role :
a) the HTTP header "Content-type:" that comes from the server, along 
with the html form
b) a possible <meta http-equiv="Content-type" ..> tag in the html 
document containing the form
c) a "Accept-charset=" attribute in the <form> tag itself

You can see (a) easily with, for instance, Firefox plus the add-on 
"LiveHTTPHeaders".
(b) and (c) can be seen with "view page source".  In Firefox, this will 
also show you what charset Firefox thinks the page is.

I am not saying that this is your problem, just that it could be, and 
that you should make 100% sure of the above elements before you look 
further in Tomcat/Websphere problems.
A slight difference in any of the above, could cause the browser to send 
things to the server a bit differently.  There are also differences 
between browsers in that respect, IE versions in particular being often 
inconsistent.

André

---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org