You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tapestry.apache.org by Denis Ponomarev <oz...@romsat.ua> on 2003/05/27 16:06:28 UTC
Non-latin charset in JavaScript messages
I am using Script component. There is the function which asks user confirmation.
Script has parameter (input-symbol) "confirmationMessage". I bind it by
<string-binding... to the message from page's *.properties file.
Everything was OK until I have created localized version of
*.properties file.
Page rendered in utf-8, but embeded javascript does not.
So non-latin literals in javascript leads to errors.
How can I fix it?
Re: Non-latin charset in JavaScript messages
Posted by Denis Ponomarev <oz...@romsat.ua>.
DP> Page rendered in utf-8, but embeded javascript does not.
DP> So non-latin literals in javascript leads to errors.
DP> How can I fix it?
Hello!
It's me again!
Walking through the source I found in the Body class
writeScript() method, which outputs scripts.
Here is two shady strings:
...
_outerWriter.printRaw(_bodyScript.toString());
...
_outerWriter.printRaw(_initializationScript.toString());
...
I think toString() is not appropriate method to render JavaScript
wich may contain non-latin messages.
JavaScript supports escape sequences for unicode.
So every non-latin character should be rendered as \u + four-digit
hexadecimal number.
I have replaced stringBuffer.toString() with call to self-implemented
method which returns string where non-latin characters has been
replaced with their escape sequences and this solves my problem.
I think something like this should be implemented in the framework
to avoid non-latin-character-in-the-script problem.
Here is my method:
private String getSafeScriptString(StringBuffer buf){
StringBuffer safe = new StringBuffer();
int length = buf.length();
for (int i = 0; i < length; i++) {
char c = buf.charAt(i);
if(c<128){
safe.append(c);
}else{
String code = ("000" + Integer.toHexString(c));
code = code.substring(code.length() - 4);
safe.append("\\u" + code );
}
}
return safe.toString();
}
Re[8]: Non-latin charset in JavaScript messages
Posted by Denis Ponomarev <oz...@romsat.ua>.
AG> What are you using as a server, Tomcat, Websphere, etc?
AG> In Tomcat, you could simply create the directory structure
AG> org/apache/tapestry/valid under common/classes in the tomcat directory and
AG> put the language specific files there.
At now I'm using JBoss with Jetty, but going to download newer
version of JBoss with Tomcat.
Thank you.
RE: Re[6]: Non-latin charset in JavaScript messages
Posted by Adam Greene <ag...@romulin.com>.
What are you using as a server, Tomcat, Websphere, etc?
In Tomcat, you could simply create the directory structure
org/apache/tapestry/valid under common/classes in the tomcat directory and
put the language specific files there.
-----Original Message-----
From: Denis Ponomarev [mailto:oz@romsat.ua]
Sent: Wednesday, August 27, 2003 11:19 AM
To: Tapestry users
Subject: Re[6]: Non-latin charset in JavaScript messages
>> I am using 3.0.
MB> I should have specified, I guess -- please use 3.0 beta-2. The earlier
MB> releases of 3.0 do not have the encoding facilities.
In 3.0 beta-2 everything works fine! Thanks a lot!
And last question on this topic - how can I add my localized version
of the ValidationStrings.properties file?
I understand that I can bind messasges from my Page.properties file to
the message-properties of validators, but this is inconvenience to do
it over all my application.
I found this in the BaseValidator class:
ResourceBundle strings =
ResourceBundle.getBundle("org.apache.tapestry.valid.ValidationStrings",
locale);
So default class loader used to load strings. As I understand it can't
see my application's classpath, isn't it?
Any suggestions?
---------------------------------------------------------------------
To unsubscribe, e-mail: tapestry-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: tapestry-user-help@jakarta.apache.org
Re[6]: Non-latin charset in JavaScript messages
Posted by Denis Ponomarev <oz...@romsat.ua>.
>> I am using 3.0.
MB> I should have specified, I guess -- please use 3.0 beta-2. The earlier
MB> releases of 3.0 do not have the encoding facilities.
In 3.0 beta-2 everything works fine! Thanks a lot!
And last question on this topic - how can I add my localized version
of the ValidationStrings.properties file?
I understand that I can bind messasges from my Page.properties file to
the message-properties of validators, but this is inconvenience to do
it over all my application.
I found this in the BaseValidator class:
ResourceBundle strings =
ResourceBundle.getBundle("org.apache.tapestry.valid.ValidationStrings", locale);
So default class loader used to load strings. As I understand it can't
see my application's classpath, isn't it?
Any suggestions?
RE: Re[4]: Non-latin charset in JavaScript messages
Posted by Mind Bridge <mi...@yahoo.com>.
Hi Denis,
> I am using 3.0.
I should have specified, I guess -- please use 3.0 beta-2. The earlier
releases of 3.0 do not have the encoding facilities.
With that version you could try the following:
Run the Workbench.
Open a new IE window, go to Tools/Internet Options/Languages, add 'Chinese
(Taiwan)' (zh-tw) as a language and place it at the top of the language list
(with highest priority). I am choosing this language as a demo, since
Tapestry contains ValidationStrings translated into Chinese (Taiwanese) by
default.
Open the Workbench in that browser, go to the Fields page, and without
typing anything (leaving client-side validation enabled), click on Continue.
What do you see?
I personally see an error message (field is required) in Chinese on a number
of different machines. This message is in the JavaScript of the page, and
thus it has to have been encoded properly in utf-8.
This is similar to what you were suggesting, I believe, and it does seem to
work consistently.
> So to force browser
> understand them you should specify charset of the page - by http
> content-type header or by http-equiv in the metatag.
3.0 beta 2 does the following:
- always sets the content-type in the header with the encoding used
- the Shell component includes http-equiv by default as well again including
the encoding (this is necessary for forms in IE)
- decodes the POST requests using that same encoding
What you are seeing is definitely very weird. I cannot think of a specific
reason why it should occur. The 'usual suspects' are the following (but they
don't quite fit what you are saying:
- the property file with the messages has not been run through
native2ascii -- this is a requirement of the standard Java Properties
implementation. As a result, the
- getResponseWriter() was overriden incorrectly in the page
I am not sure if this helps. If it does not, please contact me directly --
if this is a problem, we should get to the bottom of this.
Best regards,
-mb
-----Original Message-----
From: Denis Ponomarev [mailto:oz@romsat.ua]
Sent: Wednesday, August 27, 2003 11:22 AM
To: Tapestry users
Subject: Re[4]: Non-latin charset in JavaScript messages
>> I am ready to add, but don't know where I can do it :)
MB> :)
MB> There is a link to the bug database on jakarta.apache.org
Probably I'll try it later...
>> Of course toString() returns strings in Unicode but resulting
>> HTML contains one-byte characters, isn't it?
MB> Based on this and the rest of your message I can conclude that you are
using
MB> Tapestry 2.x.
No!
I am using 3.0. It seems like my previous explanations was not clear
enough :)
When I say "one-byte characters in HTML", I mean that when page
rendered you obtain HTML where no unicode, there are ASCII characters
only! Any non-ASCII characters are represented with special form named
"HTML entity". It consists of _several_ASCII_characters_ and looks like
Г. It is not unicode! There is several ASCII chars instead of
one non-ASCII. Of course this ASCII chars describe non-ASCII chracter
by it's unicode value, but this is not unicode exactly - this is unicode
presentation, in the case of Г it uses 7 bytes (one per char),
not 2 bytes as unicode do.
But there is no similar technic applied during javascript rendering.
MB> Version 3.0 has a number of features that, I believe, resolve the
MB> encoding/internationalization problem completely.
I don't think so!
Consider simple example:
Try to create a page with a simple form and put one ValidField on it.
Assign StringValidator to it. Set clientScriptingEnabled property to
true. (This will generate validation javascript.)
Now assume that application should have multilanguage support.
So you should specify localized versions of validation messages
for langugages you want to support. For example you can bind
requiredMessage property of the validator to the message from the
Page.properties file. To support another languages you simply add
localized properties files, for example Page_ru.properties.
Try to do it for any langugage with non-latin characters.
Now if you'll run application and open resulting HTML you'll see in the
source that validation script contains string literals with messages.
They are non-ASCII! So to force browser
understand them you should specify charset of the page - by http
content-type header or by http-equiv in the metatag.
Even if you are supporting ONE langugage and it is non-latin you
must add servlet filter to specify request encoding if you want
receive valid user input from the forms.
But we want to support _several_ languages, we can't just substitute
content-type of the output because it must support _all_ our langugages and
must have universal charset (utf-8).
So what can we do with Tapestry from this point? My answer is nothing.
And my proposed solution was quite obvious: change rendering of the
javascript. Such as it was done with HTML rendering (substituting of
non-latin with html-entities) it must be done with javascript
rendering (substituting of non-latin with [\u + code] sequences).
After doing so we don't need to specify charset other than utf-8. Our pages
will
contain ASCII characters only!
---------------------------------------------------------------------
To unsubscribe, e-mail: tapestry-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: tapestry-user-help@jakarta.apache.org
Re[4]: Non-latin charset in JavaScript messages
Posted by Denis Ponomarev <oz...@romsat.ua>.
>> I am ready to add, but don't know where I can do it :)
MB> :)
MB> There is a link to the bug database on jakarta.apache.org
Probably I'll try it later...
>> Of course toString() returns strings in Unicode but resulting
>> HTML contains one-byte characters, isn't it?
MB> Based on this and the rest of your message I can conclude that you are using
MB> Tapestry 2.x.
No!
I am using 3.0. It seems like my previous explanations was not clear
enough :)
When I say "one-byte characters in HTML", I mean that when page
rendered you obtain HTML where no unicode, there are ASCII characters
only! Any non-ASCII characters are represented with special form named
"HTML entity". It consists of _several_ASCII_characters_ and looks like
Г. It is not unicode! There is several ASCII chars instead of
one non-ASCII. Of course this ASCII chars describe non-ASCII chracter
by it's unicode value, but this is not unicode exactly - this is unicode
presentation, in the case of Г it uses 7 bytes (one per char),
not 2 bytes as unicode do.
But there is no similar technic applied during javascript rendering.
MB> Version 3.0 has a number of features that, I believe, resolve the
MB> encoding/internationalization problem completely.
I don't think so!
Consider simple example:
Try to create a page with a simple form and put one ValidField on it.
Assign StringValidator to it. Set clientScriptingEnabled property to
true. (This will generate validation javascript.)
Now assume that application should have multilanguage support.
So you should specify localized versions of validation messages
for langugages you want to support. For example you can bind
requiredMessage property of the validator to the message from the
Page.properties file. To support another languages you simply add
localized properties files, for example Page_ru.properties.
Try to do it for any langugage with non-latin characters.
Now if you'll run application and open resulting HTML you'll see in the
source that validation script contains string literals with messages.
They are non-ASCII! So to force browser
understand them you should specify charset of the page - by http
content-type header or by http-equiv in the metatag.
Even if you are supporting ONE langugage and it is non-latin you
must add servlet filter to specify request encoding if you want
receive valid user input from the forms.
But we want to support _several_ languages, we can't just substitute
content-type of the output because it must support _all_ our langugages and
must have universal charset (utf-8).
So what can we do with Tapestry from this point? My answer is nothing.
And my proposed solution was quite obvious: change rendering of the
javascript. Such as it was done with HTML rendering (substituting of
non-latin with html-entities) it must be done with javascript
rendering (substituting of non-latin with [\u + code] sequences).
After doing so we don't need to specify charset other than utf-8. Our pages will
contain ASCII characters only!
RE: Re[2]: Non-latin charset in JavaScript messages
Posted by Mind Bridge <mi...@yahoo.com>.
Hi,
> I am ready to add, but don't know where I can do it :)
:)
There is a link to the bug database on jakarta.apache.org
> Of course toString() returns strings in Unicode but resulting
> HTML contains one-byte characters, isn't it?
Based on this and the rest of your message I can conclude that you are using
Tapestry 2.x.
Version 3.0 has a number of features that, I believe, resolve the
encoding/internationalization problem completely. From the 'What's new' page
- The character encoding used for a component template can now be defined
using the property 'org.apache.tapestry.template-encoding'. The property is
localizable, so you can define 'org.apache.tapestry.template-encoding_ru' to
specify the encoding for all Russian templates, for example.
- The character encoding used to generate the response can now be specified
using the property 'org.apache.tapestry.output-encoding'. It is UTF-8 by
default. (that should not normally need to change)
- The Shell component now defines an http-equiv tag with the content type of
the response.
In other words, even without changing the default settings, the generated
HTML is encoded in UTF-8, form parameters are decoded correctly, etc.
You could separately define what encoding your templates use. That encoding
can be different for different templates (one can be ISO..., another Cp...,
a third Big5, etc), and that has no bearing at all on the charset used to
generate the final page.
Here are some ways to define the template-encoding:
<property name="org.apache.tapestry.template-encoding" value="ISO-8859-1"/>
<property name="org.apache.tapestry.template-encoding_ru"
value="windows-1251"/>
You can place those in a .jwc, .page, .library, .application, depending on
the scope in which you want them to be active. You can also define them like
other properties in the web.xml or using -D.
Please try 3.0 if you can -- you should not get one-byte chars in the output
for non-English symbols and the JavaScript should work fine.
> I hope my English was understandable :)
It was, certainly.
Best regards,
-mb
-----Original Message-----
From: Denis Ponomarev [mailto:oz@romsat.ua]
Sent: Tuesday, August 26, 2003 5:05 PM
To: Tapestry users
Subject: Re[2]: Non-latin charset in JavaScript messages
MB> Please add a bug with this, and I will make sure it is fixed. Also, will
it
MB> be possible to attach a quick example to the bug entry if possible?
(files
MB> in KOI8-R are okay -- they will make a good test).
I am ready to add, but don't know where I can do it :)
MB> I have to admit this is a bit strange, though. toString() should return
the
MB> string in Unicode, and printRaw() should output it in the encoding of
the
MB> page. Is JavaScript not accepting characters in the page encoding? (it
MB> should, I think). What exactly do you see?
Of course toString() returns strings in Unicode but resulting
HTML contains one-byte characters, isn't it?
When Tapestry generates pages all non-latin characters replaced
with well-known Ӓ stuff. Browser accept it in any charset.
But when javascript generated printRaw() used which does not do any
magic - it simply prints string. In this case I suppose that default
platform charset used to translate unicode to one-byte characters
in the resulting HTML. (Server charset, not client!)
So when client browser accepts header with information about charset
(default - utf-8) it does not know how to interpret my characters with
codes more than 127. In my case on alert("non-latin") it outputs something
like hieroglyphs :) or simply fails after page loading with "not closed
literal"
message.
Of course I can substitute content-type by overriding
getResponseWriter() method.
public IMarkupWriter getResponseWriter(OutputStream out) {
return new HTMLWriter("text/html; charset=windows-1251", out);
}
After that browser succefully accepts cyrilic.
But doing so brokes application internationality!
I can't provide support for both cyrilic and japanese anymore.
So to be charset independent I propose translate non-latin characters into
[\u + hexCode] sequences for javascript.
I hope my English was understandable :)
---------------------------------------------------------------------
To unsubscribe, e-mail: tapestry-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: tapestry-user-help@jakarta.apache.org
Re[2]: Non-latin charset in JavaScript messages
Posted by Denis Ponomarev <oz...@romsat.ua>.
MB> Please add a bug with this, and I will make sure it is fixed. Also, will it
MB> be possible to attach a quick example to the bug entry if possible? (files
MB> in KOI8-R are okay -- they will make a good test).
I am ready to add, but don't know where I can do it :)
MB> I have to admit this is a bit strange, though. toString() should return the
MB> string in Unicode, and printRaw() should output it in the encoding of the
MB> page. Is JavaScript not accepting characters in the page encoding? (it
MB> should, I think). What exactly do you see?
Of course toString() returns strings in Unicode but resulting
HTML contains one-byte characters, isn't it?
When Tapestry generates pages all non-latin characters replaced
with well-known Ӓ stuff. Browser accept it in any charset.
But when javascript generated printRaw() used which does not do any
magic - it simply prints string. In this case I suppose that default
platform charset used to translate unicode to one-byte characters
in the resulting HTML. (Server charset, not client!)
So when client browser accepts header with information about charset
(default - utf-8) it does not know how to interpret my characters with
codes more than 127. In my case on alert("non-latin") it outputs something
like hieroglyphs :) or simply fails after page loading with "not closed literal"
message.
Of course I can substitute content-type by overriding
getResponseWriter() method.
public IMarkupWriter getResponseWriter(OutputStream out) {
return new HTMLWriter("text/html; charset=windows-1251", out);
}
After that browser succefully accepts cyrilic.
But doing so brokes application internationality!
I can't provide support for both cyrilic and japanese anymore.
So to be charset independent I propose translate non-latin characters into
[\u + hexCode] sequences for javascript.
I hope my English was understandable :)
RE: Non-latin charset in JavaScript messages
Posted by Mind Bridge <mi...@yahoo.com>.
Hi Denis,
Please add a bug with this, and I will make sure it is fixed. Also, will it
be possible to attach a quick example to the bug entry if possible? (files
in KOI8-R are okay -- they will make a good test).
I have to admit this is a bit strange, though. toString() should return the
string in Unicode, and printRaw() should output it in the encoding of the
page. Is JavaScript not accepting characters in the page encoding? (it
should, I think). What exactly do you see?
Best regards,
-mb
-----Original Message-----
From: Denis Ponomarev [mailto:oz@romsat.ua]
Sent: Tuesday, August 26, 2003 2:20 PM
To: Tapestry users; Tapestry users
Subject: Re: Non-latin charset in JavaScript messages
DP> Page rendered in utf-8, but embeded javascript does not.
DP> So non-latin literals in javascript leads to errors.
DP> How can I fix it?
Hello!
It's me again!
Walking through the source I found in the Body class
writeScript() method, which outputs scripts.
Here is two shady strings:
...
_outerWriter.printRaw(_bodyScript.toString());
...
_outerWriter.printRaw(_initializationScript.toString());
...
I think toString() is not appropriate method to render JavaScript
wich may contain non-latin messages.
JavaScript supports escape sequences for unicode.
So every non-latin character should be rendered as \u + four-digit
hexadecimal number.
I have replaced stringBuffer.toString() with call to self-implemented
method which returns string where non-latin characters has been
replaced with their escape sequences and this solves my problem.
I think something like this should be implemented in the framework
to avoid non-latin-character-in-the-script problem.
Here is my method:
private String getSafeScriptString(StringBuffer buf){
StringBuffer safe = new StringBuffer();
int length = buf.length();
for (int i = 0; i < length; i++) {
char c = buf.charAt(i);
if(c<128){
safe.append(c);
}else{
String code = ("000" + Integer.toHexString(c));
code = code.substring(code.length() - 4);
safe.append("\\u" + code );
}
}
return safe.toString();
}
---------------------------------------------------------------------
To unsubscribe, e-mail: tapestry-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: tapestry-user-help@jakarta.apache.org
Re: Non-latin charset in JavaScript messages
Posted by Denis Ponomarev <oz...@romsat.ua>.
DP> Page rendered in utf-8, but embeded javascript does not.
DP> So non-latin literals in javascript leads to errors.
DP> How can I fix it?
Hello!
It's me again!
Walking through the source I found in the Body class
writeScript() method, which outputs scripts.
Here is two shady strings:
...
_outerWriter.printRaw(_bodyScript.toString());
...
_outerWriter.printRaw(_initializationScript.toString());
...
I think toString() is not appropriate method to render JavaScript
wich may contain non-latin messages.
JavaScript supports escape sequences for unicode.
So every non-latin character should be rendered as \u + four-digit
hexadecimal number.
I have replaced stringBuffer.toString() with call to self-implemented
method which returns string where non-latin characters has been
replaced with their escape sequences and this solves my problem.
I think something like this should be implemented in the framework
to avoid non-latin-character-in-the-script problem.
Here is my method:
private String getSafeScriptString(StringBuffer buf){
StringBuffer safe = new StringBuffer();
int length = buf.length();
for (int i = 0; i < length; i++) {
char c = buf.charAt(i);
if(c<128){
safe.append(c);
}else{
String code = ("000" + Integer.toHexString(c));
code = code.substring(code.length() - 4);
safe.append("\\u" + code );
}
}
return safe.toString();
}
Re[4]: Non-latin charset in JavaScript messages
Posted by "Roman \"Blade\" Velichko" <bl...@ivc.tagmet.ru>.
Hi to All!
DP> Nice hack! :)
DP> It works for me, but I think this is not appropriate approach for
DP> i18n applications and Tapestry developers should something change in
DP> javascript rendering.
May be it's a hack :-) but in own organization we have to use only
one language. Another trick for your subject is try to use
page-localization see documentations & vlib exam application...
========
With best wishes
Roman "Blade" Velichko [mailto:blade@ivc.tagmet.ru]
Re[3]: Non-latin charset in JavaScript messages
Posted by Denis Ponomarev <oz...@romsat.ua>.
Nice hack! :)
It works for me, but I think this is not appropriate approach for
i18n applications and Tapestry developers should something change in
javascript rendering.
Thank You!
DP>> Can anybody help?
DP>>> Page rendered in utf-8, but embeded javascript does not.
DP>>> So non-latin literals in javascript leads to errors.
DP>>> How can I fix it?
RBV> Try this
RBV> public class Win1251Pages extends BasePage
RBV> {
RBV> public IMarkupWriter getResponseWriter(OutputStream out)
RBV> {
RBV> return new HTMLWriter("text/html; charset=windows-1251", out);
RBV> }
RBV> }
Re[2]: Non-latin charset in JavaScript messages
Posted by "Roman \"Blade\" Velichko" <bl...@ivc.tagmet.ru>.
Hi to All!
DP> Can anybody help?
DP>> Page rendered in utf-8, but embeded javascript does not.
DP>> So non-latin literals in javascript leads to errors.
DP>> How can I fix it?
Try this
import org.apache.tapestry.*;
import org.apache.tapestry.html.*;
import java.io.*;
/**
* <p><strong>Title:</strong> WebCards</p>
* <p><strong>Description:</strong> Base class for all pages in
* application</p>
* <p><strong>Copyright:</strong> Copyright (c) 2003</p>
* <p><strong>Company:</strong> OASU-ASUP TAGMET</p>
* @author <a href="mailto:blade@ivc.tagmet.ru">Roman "Blade" Velichko</a>
*/
public class Win1251Pages extends BasePage
{
/** Use concrete code page for youee pages
* @param out
*/
public IMarkupWriter getResponseWriter(OutputStream out)
{
return new HTMLWriter("text/html; charset=windows-1251", out);
}
}
than
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE page-specification PUBLIC
"-//Howard Lewis Ship//Tapestry Specification 1.3//EN"
"http://tapestry.sf.net/dtd/Tapestry_1_3.dtd">
<page-specification class="Win1251Pages">
<component id="test" type="Insert">
<binding name="value" expression="visit.testLogin"/>
</component>
<component id="homePage" type="PageLink">
<static-binding name="page">Home</static-binding>
</component>
</page-specification>
========
With best wishes
Roman "Blade" Velichko [mailto:blade@ivc.tagmet.ru]
Re: Non-latin charset in JavaScript messages
Posted by Denis Ponomarev <oz...@romsat.ua>.
Can anybody help?
DP> Page rendered in utf-8, but embeded javascript does not.
DP> So non-latin literals in javascript leads to errors.
DP> How can I fix it?