character encodingg problem

["Russo, Joe" <Ru...@cadmus.com>]

Hi,
This information you provided helped me a lot.  I was able to include
the filter and get the correct data.  The main problem I am having is
the current data is basically corrupt.  

This is what I believe is happening:  some of the data is having the
wrong character encoding into the database and causes those funky
characters.  If I include the filter it does work with the existing data
being encoded properly and no bad or correct data is put into the DB.

Of course, the existing data that is stored incorrectly can not be
viewed correctly.  I thought by changing the filter you sent by
interrupting the charset as ISO-8859-1 for the response would work
correctly.  This would be great if it work that it would have all new
data put in correctly and all the old data would be seen correctly.  

Any ideas or help would be appreciated.
Joe

import javax.servlet.*;
import java.io.*;
import javax.servlet.http.*;

public class ContentTypeFilter
        implements Filter{

  public void init(FilterConfig config){}
  public void destroy() {}
  public void doFilter(ServletRequest request, ServletResponse response,
        FilterChain filterChain) throws IOException, ServletException{
    request = (HttpServletRequest)request;
    request.setCharacterEncoding("UTF-8");
    response.setCharacterEncoding("ISO-8859-1");
    response.setContentType("text/html;charset=ISO-8859-1");

    filterChain.doFilter(request, response);
  }
}


-----Original Message-----
From: Nathan Hook [mailto:hooknc@hotmail.com] 
Sent: Wednesday, July 25, 2007 2:13 PM
To: users@tomcat.apache.org
Subject: Re: Tomcat5.0.28 character encodingg problem

Both Chris and Tim are giving great advice.  We're actually just trying
to 
internationalize our application for our next major release.

Here are the things we've learned.

- You have to change the URIEncoding on your Tomcat Connector in your 
server.xml (as Tim pointed out).

We are using mod_jk and had to change our entry in the server.xml to the

following:

<Connector port="8009"
           enableLookups="false" redirectPort="8443" protocol="AJP/1.3" 
URIEncoding="UTF-8" />


- On every request that comes into your tomcat server you have to check
the 
character encoding of your request and your response BEFORE any work is 
actually done.

So like Chris mentioned you want to look up a character encoding filter.
I 
would recommend placing that as the very first filter that gets called
in 
your application.  To do make this filter first in the filter chain is 
simple.  When adding your filter to your applications web.xml file, make

sure is the first one listed in the filter mappings section.

Here is the Filter we are currently using for testing.

public class ContentTypeFilter implements Filter {
  public void init(FilterConfig config) {}
  public void destroy() {}
  public void doFilter(ServletRequest request, ServletResponse response,

FilterChain filterChain) throws IOException, ServletException
  {
     // I've seen some other classes that check to see if the character 
encoding is null and then set
     // the character encoding to utf-8.  I'm not sure which is best at
this 
time.  My guess is doing
     // the null checks because from my understanding the client can
change 
the page encoding on
     // each and every request even though the server sets the page up
to be 
utf-8.
     request = (HttpServletRequest)request;
     request.setCharacterEncoding("UTF-8");

     // Make sure to set the character encoding on the response early 
because once something is
     // sent back to the client (like a jsp), then the character
encoding is 
already set to the default
     // of the server.
     response.setCharacterEncoding("UTF-8");
     // Set the content type in the header of the response.
     response.setContentType("text/html;charset=UTF-8");

     filterChain.doFilter(request, response);
  }
}


- Set the meta type in each and every jsp to be utf-8.  Now, most
browsers 
will ignore this value from my understanding, but it shouldn't hurt to
add 
it.

<head>
  <title>test title</title>
  <meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
</head>


- Finally for database storage...  Again from my understanding you will
need 
to set all your tables to utf-8 and then inform your JDBC Driver that
you 
want to pass everything back and forth using utf-8.

In mysql you add the following to your jdbc url connection string:
useUnicode=true
characterEncoding=UTF-8


I hope all that information helps.


----Original Message Follows----
From: Tim Funk <fu...@joedog.org>
Reply-To: "Tomcat Users List" <us...@tomcat.apache.org>
To: Tomcat Users List <us...@tomcat.apache.org>
Subject: Re: Tomcat5.0.28 character encodingg problem
Date: Wed, 25 Jul 2007 12:09:07 -0400

http://tomcat.apache.org/faq/misc.html#utf8

And you should first start with in server.xml:
     <Connector ... URIEncoding="UTF-8" .../>

-Tim

Joe Russo wrote:
>I am getting the following error in the display of the JSP.  To give a
>little history, this application I am supporting, at the time the
>developers thought they needed to encode the characters to UTF-8 into
>our Oracle DB.  The developers were unaware they could have allowed the
>DB Driver convert it for us.  Therefore, we double encode going into
and
>out of the database.  Really stupid in hindsight.  Trying to clean the
>database up is another project we face.
>
>I am in the process of converting from using JRUN to Tomcat and I have
>ran into the problem where these funky symbols are displaying.  I can
>not find any stack traces that would explain or possibly clue into a
>solution.
>
>My questions are:  Does Tomcat have problems with any types of
encoding?    
>   What type of characters are being displayed below and any advice in
>troubleshooting or solving this would be gratefully appreciated.
>

---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

_________________________________________________________________
http://newlivehotmail.com


---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: character encodingg problem

[Christopher Schultz <ch...@christopherschultz.net>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Joe,

Russo, Joe wrote:
> Of course, the existing data that is stored incorrectly can not be
> viewed correctly.  I thought by changing the filter you sent by
> interrupting the charset as ISO-8859-1 for the response would work
> correctly.

No, the problem is that your application thinks that the data in the DB
is in UTF-8 format, but it's not. Here's what you have to do:

1.  Identify the rows that need to be "fixed".
2a. Run a query that reads the data as bytes (not characters), and then
    cast it to UTF-8 character data, then write it back.
2b. Write a little Java program to read the data as bytes (not chars),
    convert to a String using the suspected encoding (UTF-8, right?),
    and then writing it back to the database.

This ought to work, but might be kind of a pain in the neck. Of course,
always test that things are working the way you expect them to before
you COMMIT anything ;)

I would recommend using a Java program. Write a quick test like this:

SELECT my_data ... WHERE ... FOR UPDATE

ResultSet rs = ...;
InputStream in = rs.getBinaryStream("my_data");
int count = 0;
String s = "";
byte[] bytes = new byte[1024];

while(0 <= (c = in.read(bytes))
    s = s + new String(bytes, 0, c, "ISO-8859-1"); // or whatever

System.out.println("Read string from database: " + s);

rs.updateString("my_data", s);

rs.close();

SELECT my_data ... WHERE ...

s = rs.getString("my_data");

System.out.println("Read fixed string from database: " + s);

conn.rollback();

Make sure that your terminal is configured to properly display UTF-8
character data, or that you have some other good way of determining that
the test output is correct.

- -chris

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFGwKpM9CaO5/Lv0PARAnjeAKCbLF9KK2gkaVTE0yjbs6YMPRfFUACfUjAR
SIPO2gjeAzdwK8iUreSF3uY=
=FKXu
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org