You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Nikola Milutinovic <Ni...@ev.co.yu> on 2001/12/04 15:23:34 UTC
Character Encoding problems
Hi all.
I'm developing a web application that uses textual data for Central&Eastern Europe. The text is in a database which is internally UNICODE. I also have another DB instance which is ISO-8859-2 encoded, so all options are in the play.
I thought I should set contentType="text/html; charset=ISO-8859-2", declare a page to the same (just in case) and "sit back and enjoy myself". Unfortunately, I was wrong.
Not only is the Latin-2 support in both IE and Netscape buggy (they wouldn't display "s-caron" and "z-caron", but would display Caps versions of those characters), but Java is bugging me, too. Instead of letters specific to our alphabet, I'm getting "?".
With the help of a dedicated PostgreSQL JDBC developer, I have tracked this problem down to JVM, which has a default encoding of "ISO-8859-1". In a standalone Java application I can do explicit encoding, like this:
System.out.write( testString.getBytes( "ISO-8859-2" ) );
and it will print the characters I expect, instead of "?".
What do I do in Tomcat?
I have set contentType to "text/html; charset=ISO-8859-2" and in a generated Servlet code it really has:
response.setContentType("text/html; charset=ISO-8859-2");
So, no trouble there. How do I get a (Unicode) string to convert to a ISO-8859-2 encoded byte stream? Because, eventually, that is what the browser should get. I cannot use the method from above, since JspWriter doesn't accept byte[] as an argument.
Nix.
Re: Character Encoding problems
Posted by dimiter <di...@herti.bg>.
I was in the same situation. But look the attached file and you'll have a
way to set any kind of character set.
Dimiter
Re: Character Encoding problems
Posted by Nikola Milutinovic <Ni...@ev.co.yu>.
Ivo Panacek wrote:
>> Hmmm, since I'm in a JSP page and not Servlet, this should be done
>> automagically. When I run Jasper on the "test.jsp" file, this is what I
>
>
> Aha. I use JSP only too and it works for me.
> My configuration is:
>
> RedHat 7.1 + Apache + jdk 1.3 (blackdown) + Tomcat 4.0.1 + webapp module.
>
> This was my testing page:
>
> <%--
> <%@ page contentType="text/html;charset=ISO-8859-2" %>
> <%@ page pageEncoding="ISO-8859-2" %>
> --%>
> <%@ page contentType="text/html;charset=UTF-8" %>
> <%@ page pageEncoding="UTF-8" %>
>
> <%@ page import="HelloBean3" %>
>
> <jsp:useBean id="mojeAhojFazole3" class="HelloBean3" scope="session"/>
> <jsp:setProperty name="mojeAhojFazole3" property="*"/>
>
> <html>
> <head><title>mojeAhojFazole3</title></head>
> <body bgcolor="white" onLoad="focusInput()">
>
> <p>
> Gratuluji, tohle opravdu je funkc(ní JSP aplikace,
> která dokonce ne(co de(lá.
> </p>
>
> ... form testing
Similar stuff here, I just stuff all "page" directives into one, but it sets all
it should set in a resulting Java file.
The trouble is, I've tested a Servlet, too. Again, a "failure to communicate"
occurs - I get "?". The same ones I get when I retrieve data via JDBC and
display it with default JVM encoding.
> Both UTF-8 and ISO-8859-2 versions worked well,
> tested with Mozilla (0.9.x) on Linux and IE on Windows.
They should.
> ISO-8859-2 characters were in html text, but relevant
> part of resulting java is:
>
> out.write("\r\n\r\n<html>\r\n<head><title>mojeAhojFazole3</title></head>\r\n<body
> bgcolor=\"white\">\r\n\r\n<p>\r\nGratuluji, tohle opravdu je funkÄnA(
> JSP aplikace,\r\nkterA(A; dokonce nÄ?co
> dÄ?lA(A;.\r\n</p>\r\n<p><hr></p>\r\n\r\n");
>
> ... so simple out.write, text is in default UTF-8.
So, Jasper transformed ISO-8859-2 to UTF-8? It never did that for me...
> Problem was only with input from forms. Testing showed, that
> browser does not write encoding in mime headers, but it use
> the same encoding as in original page. So I use simple
> filter (found via this mail-list -- name was
> SetCharacterEncodingFilter.java -- I can send it)
> and now I have no problems.
I still haven't come that far :-<
>> %>
>> <%!
>> String testText;
>> %>
>> <%
>> testText = "\uC5A0 \uC5A1 \uC486 \uC487 \uC48C \uC48D \uC490 \uC491
>> \uC5BD \uC5BE";
>> %>
>
> Just one idea:
> what are those unicode characters?
Those are Unicode characters for: S-caron, s-caron, C-acsan, c-acsan, C-caron,
c-caron, D-slash, d-slash, Z-caron and z-caron.
> I've just tested this code:
>
> public static void main( String args[] ) {
> String text1 = "\uC5A0 \uC5A1 \uC486 \uC487 \uC48C \uC48D \uC490
> \uC491 \uC5BD \uC5BE";
> String text2 = "e(s(c(r(z(ýáíé";
>
> try {
> PrintWriter wr = new PrintWriter(
> new OutputStreamWriter( System.out, "8859_2" ) );
> wr.println("\""+text1+"\"");
> wr.println("\""+text2+"\"");
> wr.close();
> }
> catch( Exception e ) {
> System.err.println("Error: "+e);
> }
> }
> }
>
>
> And result is:
>
> "? ? ? ? ? ? ? ? ? ?"
> "e(s(c(r(z(ýáíé"
The trouth is, a JDBC driver expert from PostgreSQL said the only way to produce
correct output with my test string was to use:
wr.write( text1.getBytes( "ISO-8859-2" ) );
I'm quite sure that I'm not messing anything up. Or if I am, then so is
PostgreSQL. I have entered Win-1250 data into a database, which had an
ISO-8859-2 internal encoding. I was using correct client-to-server encoding and
the data ended up as ISO-8859-2 formated strings. Then I used JDBC to access the
data and I really got UTF-8 with our alphabet specific characters. Only
converting that string to a byte array with specific encoding would do the trick.
I'll try UTF-8 next time.
Nix.
--
To unsubscribe: <ma...@jakarta.apache.org>
For additional commands: <ma...@jakarta.apache.org>
Troubles with the list: <ma...@jakarta.apache.org>
Re: Character Encoding problems
Posted by Ivo Panacek <Iv...@jlabs.cz>.
Nikola Milutinovic wrote:
> Ivo Panacek wrote:
>
>>> response.setContentType("text/html; charset=ISO-8859-2");
>>>
>>> So, no trouble there. How do I get a (Unicode) string to convert to a
>>> ISO-8859-2 encoded byte stream? Because, eventually, that is what the
>>> browser should get. I cannot use the method from above, since
>>> JspWriter doesn't accept byte[] as an argument.
>>
>>
>>
>> Java does it for you. If you retrieve output stream AFTER setting
>> content type
>> with out = pageContext.getOut(); you write to out in UNICODE and output
>> is in ISO-8859-2.
>
>
> Hmmm, since I'm in a JSP page and not Servlet, this should be done
> automagically. When I run Jasper on the "test.jsp" file, this is what I
Aha. I use JSP only too and it works for me.
My configuration is:
RedHat 7.1 + Apache + jdk 1.3 (blackdown) + Tomcat 4.0.1 + webapp module.
This was my testing page:
<%--
<%@ page contentType="text/html;charset=ISO-8859-2" %>
<%@ page pageEncoding="ISO-8859-2" %>
--%>
<%@ page contentType="text/html;charset=UTF-8" %>
<%@ page pageEncoding="UTF-8" %>
<%@ page import="HelloBean3" %>
<jsp:useBean id="mojeAhojFazole3" class="HelloBean3" scope="session"/>
<jsp:setProperty name="mojeAhojFazole3" property="*"/>
<html>
<head><title>mojeAhojFazole3</title></head>
<body bgcolor="white" onLoad="focusInput()">
<p>
Gratuluji, tohle opravdu je funkc(ní JSP aplikace,
která dokonce ne(co de(lá.
</p>
... form testing
Both UTF-8 and ISO-8859-2 versions worked well,
tested with Mozilla (0.9.x) on Linux and IE on Windows.
ISO-8859-2 characters were in html text, but relevant
part of resulting java is:
out.write("\r\n\r\n<html>\r\n<head><title>mojeAhojFazole3</title></head>\r\n<body bgcolor=\"white\">\r\n\r\n<p>\r\nGratuluji, tohle opravdu je funkÄnA( JSP
aplikace,\r\nkterA(A; dokonce nÄ?co dÄ?lA(A;.\r\n</p>\r\n<p><hr></p>\r\n\r\n");
... so simple out.write, text is in default UTF-8.
Problem was only with input from forms. Testing showed, that
browser does not write encoding in mime headers, but it use
the same encoding as in original page. So I use simple
filter (found via this mail-list -- name was
SetCharacterEncodingFilter.java -- I can send it)
and now I have no problems.
> %>
> <%!
> String testText;
> %>
> <%
> testText = "\uC5A0 \uC5A1 \uC486 \uC487 \uC48C \uC48D \uC490 \uC491 \uC5BD \uC5BE";
> %>
Just one idea:
what are those unicode characters ?
I've just tested this code:
public static void main( String args[] ) {
String text1 = "\uC5A0 \uC5A1 \uC486 \uC487 \uC48C \uC48D \uC490 \uC491 \uC5BD \uC5BE";
String text2 = "e(s(c(r(z(ýáíé";
try {
PrintWriter wr = new PrintWriter(
new OutputStreamWriter( System.out, "8859_2" ) );
wr.println("\""+text1+"\"");
wr.println("\""+text2+"\"");
wr.close();
}
catch( Exception e ) {
System.err.println("Error: "+e);
}
}
}
And result is:
"? ? ? ? ? ? ? ? ? ?"
"e(s(c(r(z(ýáíé"
ivo
--
E-mail: Ivo.Panacek@regionet.cz, Ivo.Panacek@jlabs.cz
WWW: http://ivop.regionet.cz
Mobile: +420 602 337776
--
To unsubscribe: <ma...@jakarta.apache.org>
For additional commands: <ma...@jakarta.apache.org>
Troubles with the list: <ma...@jakarta.apache.org>
Re: Character Encoding problems
Posted by Nikola Milutinovic <Ni...@ev.co.yu>.
Ivo Panacek wrote:
>> response.setContentType("text/html; charset=ISO-8859-2");
>>
>> So, no trouble there. How do I get a (Unicode) string to convert to a
>> ISO-8859-2 encoded byte stream? Because, eventually, that is what the
>> browser should get. I cannot use the method from above, since
>> JspWriter doesn't accept byte[] as an argument.
>
>
> Java does it for you. If you retrieve output stream AFTER setting
> content type
> with out = pageContext.getOut(); you write to out in UNICODE and output
> is in ISO-8859-2.
Hmmm, since I'm in a JSP page and not Servlet, this should be done
automagically. When I run Jasper on the "test.jsp" file, this is what I get:
TEST.JSP
--------
<%@ page
info="A test page"
contentType="text/html; charset=ISO-8859-2"
%>
<%!
String testText;
%>
<%
testText = "\uC5A0 \uC5A1 \uC486 \uC487 \uC48C \uC48D \uC490 \uC491 \uC5BD \uC5BE";
%>
<html>
<head>
<title>Test page</title>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-2">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<h1>Test page for encoding issues</h1>
<p>And this is generated: <%= testText %></p>
<p>And this is from a scriptlet: <% out.write( testText.getBytes("LATIN2") ); %>
<p>Nix.</p>
</body>
</html>
TEST.JAVA
---------
[SNIP]
response.setContentType("text/html; charset=ISO-8859-2");
pageContext = _jspxFactory.getPageContext(this, request, response,
"", true, 8192, true);
application = pageContext.getServletContext();
config = pageContext.getServletConfig();
session = pageContext.getSession();
out = pageContext.getOut();
[SNIP]
-----------
So, the "out" object IS obtained AFTER the content type has been set. Still,
both <%= testText %> and the scriptlet produce the same result - "?" instead of
Latin-2 characters.
It might be the JVM issue and I'll look into the possible options for setting
the default output encoding, tomorrow.
BTW, I'd say you're using LATIN-2, so how do browser go about it? I have tried
to create a static HTML page with what I believe is Latin-2 text (I have the
right keyboard mapping). The result is frustrating, all characters are there,
except small s-caron and small z-caron (capitals are OK). Any idea?
Nix.
--
To unsubscribe: <ma...@jakarta.apache.org>
For additional commands: <ma...@jakarta.apache.org>
Troubles with the list: <ma...@jakarta.apache.org>
Re: Character Encoding problems
Posted by Ivo Panacek <Iv...@jlabs.cz>.
> response.setContentType("text/html; charset=ISO-8859-2");
>
> So, no trouble there. How do I get a (Unicode) string to convert to a ISO-8859-2 encoded byte stream? Because, eventually, that is what the browser should get. I cannot use the method from above, since JspWriter doesn't accept byte[] as an argument.
Java does it for you. If you retrieve output stream AFTER setting content type
with out = pageContext.getOut(); you write to out in UNICODE and output
is in ISO-8859-2.
ivo
--
E-mail: Ivo.Panacek@regionet.cz, Ivo.Panacek@jlabs.cz
WWW: http://ivop.regionet.cz
Mobile: +420 602 337776
--
To unsubscribe: <ma...@jakarta.apache.org>
For additional commands: <ma...@jakarta.apache.org>
Troubles with the list: <ma...@jakarta.apache.org>
Re: webroot folder different than web app folder. possible?
Posted by amol <ma...@yahoo.com>.
> It *must* be "WEB-INF" (all upper case)
yeah..it *is* WEB-INF.
dont take the lower case in the mail soo seriously. i have typed all that
manually. -:)
any comments on the *real* problem?
the stuff is working peacefully right now on tomcat. i am new to
tomcat/apache and even this web apps specification thing and am looking for
some inputs which would help me come out with a good deployment strategy.
thnx
amol
----- Original Message -----
From: "Craig R. McClanahan" <cr...@apache.org>
To: "Tomcat Users List" <to...@jakarta.apache.org>
Sent: Wednesday, December 05, 2001 1:45 PM
Subject: Re: webroot folder different than web app folder. possible?
>
>
> On Wed, 5 Dec 2001, amol wrote:
>
> > Date: Wed, 5 Dec 2001 13:12:58 +0530
> > From: amol <ma...@yahoo.com>
> > Reply-To: Tomcat Users List <to...@jakarta.apache.org>
> > To: Tomcat Users List <to...@jakarta.apache.org>
> > Subject: webroot folder different than web app folder. possible?
> >
> > I have got the following folder structure in tomcat around my web
> > application:
> >
> > -bin
> > -webapps
> > -myapplication
> > -myapplicationwwwroot
> > -gifs
> > -index.html
> > -...etc
> > -web-inf
>
> If this is really "web-inf" (all lower case), you are doomed to failure.
> It *must* be "WEB-INF" (all upper case) instead, because servlet based
> applications are always case sensitive.
>
> > -lib
> > -classes
> > -parameters
> > -work
> > -...( etc )
> >
>
> Craig McClanahan
>
>
> --
> To unsubscribe: <ma...@jakarta.apache.org>
> For additional commands: <ma...@jakarta.apache.org>
> Troubles with the list: <ma...@jakarta.apache.org>
_________________________________________________________
Do You Yahoo!?
Get your free @yahoo.com address at http://mail.yahoo.com
--
To unsubscribe: <ma...@jakarta.apache.org>
For additional commands: <ma...@jakarta.apache.org>
Troubles with the list: <ma...@jakarta.apache.org>
Re: webroot folder different than web app folder. possible?
Posted by "Craig R. McClanahan" <cr...@apache.org>.
On Wed, 5 Dec 2001, amol wrote:
> Date: Wed, 5 Dec 2001 13:12:58 +0530
> From: amol <ma...@yahoo.com>
> Reply-To: Tomcat Users List <to...@jakarta.apache.org>
> To: Tomcat Users List <to...@jakarta.apache.org>
> Subject: webroot folder different than web app folder. possible?
>
> I have got the following folder structure in tomcat around my web
> application:
>
> -bin
> -webapps
> -myapplication
> -myapplicationwwwroot
> -gifs
> -index.html
> -...etc
> -web-inf
If this is really "web-inf" (all lower case), you are doomed to failure.
It *must* be "WEB-INF" (all upper case) instead, because servlet based
applications are always case sensitive.
> -lib
> -classes
> -parameters
> -work
> -...( etc )
>
Craig McClanahan
--
To unsubscribe: <ma...@jakarta.apache.org>
For additional commands: <ma...@jakarta.apache.org>
Troubles with the list: <ma...@jakarta.apache.org>
webroot folder different than web app folder. possible?
Posted by amol <ma...@yahoo.com>.
I have got the following folder structure in tomcat around my web
application:
-bin
-webapps
-myapplication
-myapplicationwwwroot
-gifs
-index.html
-...etc
-web-inf
-lib
-classes
-parameters
-work
-...( etc )
my problem is the following :
i want to provide a "quick installation" in which the user ( sys admin )
would use the tomcat's web server itself and not connect to any Apache or
IIS or anything like that.
but this wont be a viable solution incase the user wants to actually deploy
the application for heavy use. ( or would it be? )
in that case, he will have to connect Apache or IIS and in that, he would
need to specify the root folder for the site. my problem is that i cant make
him specify the entire 'myappliaction' folder as the root folder because the
web-inf then becomes visible to the outside world which i dont want.
and if he makes myapplicationwwwroot as the default root, my relative paths
change and he'll have to edit some parameter file for that.
one way is to make him create a "dummy" folder in which he can put the
"myapplicationwwwroot" folder. but then, the app wont work with tomcat's web
server.
is there any way to keep the folder structure same and let the user use
either tomcat's web server or apache or IIS as and when he wants?
in tomcat, can tomcat web servers root directory be explicitly specified for
a particular web application?
thnx
amol
_________________________________________________________
Do You Yahoo!?
Get your free @yahoo.com address at http://mail.yahoo.com
--
To unsubscribe: <ma...@jakarta.apache.org>
For additional commands: <ma...@jakarta.apache.org>
Troubles with the list: <ma...@jakarta.apache.org>
Re: Character Encoding problems
Posted by Nikola Milutinovic <Ni...@ev.co.yu>.
Marcin Jaskula wrote:
>>So, no trouble there. How do I get a (Unicode) string to convert
>>to a ISO-8859-2 encoded byte stream? Because, eventually, that is
>>what the browser should get. I cannot use the method from above,
>>since JspWriter doesn't accept byte[] as an argument.
> I managed it by:
> 1. The page with the input form MUST have encoding=iso-8859-2
> <meta http-equiv="content-type" content="text/html; charset=iso-8859-2">
> 2. The DB encoding LATIN-2
> 3. In the source of the jsp page:
> <%@ page
> contentType="text/html; charset=iso-8859-2"
> ....
> %>
> request.setCharacterEncoding("iso-8859-2"); // just after <%@page !! before
> you
> // read any
> argument from request
>
> and in the http headers:
> <meta http-equiv="content-type" content="text/html; charset=iso-8859-2">
> // ^^^^^ it looks unnecessary but otherwise it doesn't work ???
Still no go. I'm not even trying to POST anything yet - just GET the
data from a database. I know for sure that the data is OK and that JDBC
is giving a correct string. The problem is in Tomcat using the (system?)
default character encoding ISO-8859-1.
Would changing local settings of the OS result in a change? Is this a
JVM or Tomcat issue? I think Tomcat should convert characters.
Nix.
--
To unsubscribe: <ma...@jakarta.apache.org>
For additional commands: <ma...@jakarta.apache.org>
Troubles with the list: <ma...@jakarta.apache.org>
Re: Character Encoding problems
Posted by Nikola Milutinovic <Ni...@ev.co.yu>.
Marcin Jaskula wrote:
>
> I used to have similar problem.
> I'm using Tomcat 4.0 with PostgreSQL, DB encoding LATIN-2.
> I couldn't get proper characters from the page source, an input form (post
> method) and from the DB.
> I managed it by:
> 1. The page with the input form MUST have encoding=iso-8859-2
> <meta http-equiv="content-type" content="text/html; charset=iso-8859-2">
> 2. The DB encoding LATIN-2
> 3. In the source of the jsp page:
> <%@ page
> contentType="text/html; charset=iso-8859-2"
> ....
> %>
> request.setCharacterEncoding("iso-8859-2"); // just after <%@page !! before
> you
> // read any
> argument from request
>
> and in the http headers:
> <meta http-equiv="content-type" content="text/html; charset=iso-8859-2">
> // ^^^^^ it looks unnecessary but otherwise it doesn't work ???
Thanks.
Tomcat developers... any comment?
Nix.
--
To unsubscribe: <ma...@jakarta.apache.org>
For additional commands: <ma...@jakarta.apache.org>
Troubles with the list: <ma...@jakarta.apache.org>
RE: Character Encoding problems
Posted by Marcin Jaskula <ja...@expro.pl>.
> I'm developing a web application that uses textual data for
> Central&Eastern Europe. The text is in a database which is
> internally UNICODE. I also have another DB instance which is
> ISO-8859-2 encoded, so all options are in the play.
>
> I thought I should set contentType="text/html;
> charset=ISO-8859-2", declare a page to the same (just in case)
> and "sit back and enjoy myself". Unfortunately, I was wrong.
>
> Not only is the Latin-2 support in both IE and Netscape buggy
> (they wouldn't display "s-caron" and "z-caron", but would display
> Caps versions of those characters), but Java is bugging me, too.
> Instead of letters specific to our alphabet, I'm getting "?".
>
> With the help of a dedicated PostgreSQL JDBC developer, I have
> tracked this problem down to JVM, which has a default encoding of
> "ISO-8859-1". In a standalone Java application I can do explicit
> encoding, like this:
>
> System.out.write( testString.getBytes( "ISO-8859-2" ) );
>
> and it will print the characters I expect, instead of "?".
>
> What do I do in Tomcat?
>
> I have set contentType to "text/html; charset=ISO-8859-2" and in
> a generated Servlet code it really has:
>
> response.setContentType("text/html; charset=ISO-8859-2");
>
> So, no trouble there. How do I get a (Unicode) string to convert
> to a ISO-8859-2 encoded byte stream? Because, eventually, that is
> what the browser should get. I cannot use the method from above,
> since JspWriter doesn't accept byte[] as an argument.
Hi
I used to have similar problem.
I'm using Tomcat 4.0 with PostgreSQL, DB encoding LATIN-2.
I couldn't get proper characters from the page source, an input form (post
method) and from the DB.
I managed it by:
1. The page with the input form MUST have encoding=iso-8859-2
<meta http-equiv="content-type" content="text/html; charset=iso-8859-2">
2. The DB encoding LATIN-2
3. In the source of the jsp page:
<%@ page
contentType="text/html; charset=iso-8859-2"
....
%>
request.setCharacterEncoding("iso-8859-2"); // just after <%@page !! before
you
// read any
argument from request
and in the http headers:
<meta http-equiv="content-type" content="text/html; charset=iso-8859-2">
// ^^^^^ it looks unnecessary but otherwise it doesn't work ???
All the best
Marcin
--
To unsubscribe: <ma...@jakarta.apache.org>
For additional commands: <ma...@jakarta.apache.org>
Troubles with the list: <ma...@jakarta.apache.org>