You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@turbine.apache.org by Eric Dobbs <er...@dobbse.net> on 2002/08/30 19:40:12 UTC
Yankee needs Localization advice
Hi All.
Need some opinions about localization for web
applications. We have been getting some interest in
our product from non-US companies, and are trying to
get a sense of the work required to localize our
application. I realize this is a non-trivial problem
and that much of it will depend on the nature of our
application. Nevertheless, I'm interested your
experience supporting multiple languages in Turbine
apps.
There seem to be four aspects to the problem:
1. the display (users need to be able to read the
interface in their own language),
2. the data entry (we need to correctly encode the data
entered by users on its way into the system),
3. how the localized data gets sorted in the display
(or what the database must know about the language),
and
4. how the localized strings effect the page layout
with regard to size, direction, and orientation. (If
I remember correctly, Arabic reads right to left,
top to bottom, Chinese reads top to bottom, right to
left).
For the time being, I'm primarily interested in 1, 2,
and 3, and am choosing to ignore 4.
For the display side there seem to be two ways to solve
the problem:
A. create a separate set of templates for each locale, or
B. use the LocalizationService to enable a single set
of templates to support multiple locales.
Consequences of A are that it is less work up-front,
but more work for each language. And it also makes the
modification of the localized pages challenging for
developers not conversant in that language. Also, this
solution does not address the output in log files, or
other administrative aspects of the system, only that
which is accessible through the web interface.
Consequence of B are that it is more work up-front, but
less work for each language thereafter. And it makes
the single set of templates harder to read, and more
complex as languages are added, but perhaps easier to
maintain for language-challenged Yankee developers.
What is your experience?
For data entry, there are another couple areas of
concern: GET vs POST arguments. I know that GETs put
the arguments in the URL, and therefore must be encoded
using the usual %xx scheme. For POSTs I found this
email which looks too good to be true:
<http://nagoya.apache.org/eyebrowse/ReadMsg?listName=turbine-
dev@jakarta.apache.org&msgId=248439>
Is it really as simple as adding one of these to our
<form> tags?
<form accept-charset="UTF-8, ISO-8859-1" ...>
<form enctype="application/x-www-form-urlencoded; charset=UTF-8" ...>
I know that is what UTF-8 and Unicode are supposed to
solve. Does it really work in the real world? Does
that really handle all of the major languages?
What is your experience?
As for sorting the data...
This is an area about which I have found very little
info and even less experience or exposure.
This seems easier to describe with a scenario. Suppose
our customer is a supplier in Europe who provides parts
to manufacturers in various other countries. Their
customers can view the site in their language of
choice. And they enter data into the application in
their own language as well. The data in the database
is now in multiple languages.
How do you handle the sorting of data in multiple
languages? If you are using ORDER BY clauses in your
SQL, how does the database know which language to use
for sorting?
Is it a common practice to include a column in the
tables for a language code, so that you can SELECT only
the records for a specific language?
Or do people just avoid this problem and require their
customers to enter data in a particular language?
Or do developers just do the sorting in Java instead of
in the database?
Or would you actually put the data into separate
databases, or instances, one for each locale?
Can you tell that I have zero knowledge about this last
aspect of the problem? 8^)
Thanks for taking the time to read my questions. If
this information is already well documented somewhere,
I'll gladly accept pointers to any resources you can
offer.
-Eric
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Yankee needs Localization advice
Posted by Eric Dobbs <er...@dobbse.net>.
Hi Dan.
Thank you very much for your detailed replies and for
the pointers to O'Reilly books.
I really appreciate it.
-Eric
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Yankee needs Localization advice
Posted by Daniel Rall <dl...@finemaltcoding.com>.
Eric Dobbs <er...@dobbse.net> writes:
> Hi All.
>
> Need some opinions about localization for web
> applications. We have been getting some interest in
> our product from non-US companies, and are trying to
> get a sense of the work required to localize our
> application. I realize this is a non-trivial problem
> and that much of it will depend on the nature of our
> application. Nevertheless, I'm interested your
> experience supporting multiple languages in Turbine
> apps.
The degree of localization difficulty varies depending upon the
languages and locals you're interested in supporting. For instance,
handling only Latin-based languages is much more straight forward for
an app originally written for English than handling Japanese.
O'Reilly has a couple of great books and on this subject:
"CJKV Information Processing"
http://www.oreilly.com/catalog/cjkvinfo/
"Java Internationalization"
http://www.oreilly.com/catalog/javaint/
And one which I recommend over the others but couldn't find a
reference to. It has a picture of a globe on the front, and has
yellow lettering. It's possibly just a different printing of "Java
Internationalization" (I'd know if I could remember the title, damn
it ;).
While I was grabbing these URLs from their site, I also noticed some
articles which may be of interest. Try sticking permutations of
"internationalization", "i18n", "localization", "l10n", and
"globalization" into their search.
> There seem to be four aspects to the problem:
>
> 1. the display (users need to be able to read the
> interface in their own language),
>
> 2. the data entry (we need to correctly encode the data
> entered by users on its way into the system),
>
> 3. how the localized data gets sorted in the display
> (or what the database must know about the language),
> and
>
> 4. how the localized strings effect the page layout
> with regard to size, direction, and orientation. (If
> I remember correctly, Arabic reads right to left,
> top to bottom, Chinese reads top to bottom, right to
> left).
>
> For the time being, I'm primarily interested in 1, 2,
> and 3, and am choosing to ignore 4.
>
>
> For the display side there seem to be two ways to solve
> the problem:
>
> A. create a separate set of templates for each locale, or
Try hard to avoid creating separate templates. Whether you want them
to or not, your templates are bound to contain logic (probably
display-centric, but logic none the less). Duplicating this logic
across multiple copies of templates leading to a maintainence
nightmare.
On the flip side, for static content like HTML, this is exactly what I
recommend doing. For a web application, use an Apache MultiViews
suffix (i.e. .en, .ja, etc.) for each copy, and use Apache httpd's
MultiViews and/or Jakarta Velocity's servlet tools for selecting the
right file (I base my selection algorithm around the user's
Accept-Lange HTTP header). Employing this method for static content
cuts down on the maintainence overhead of resource bundles/message
catalogs/whatever ya wanna call'em.
> B. use the LocalizationService to enable a single set
> of templates to support multiple locales.
For any display template which contain or may contain any sort of
logic or flow control, this is clearly the way to go. Be sure to use
something like LocalizationService/LocalizationTool which allows you
to plug in an alternate translation selection algorithm in case the
default is not to your liking or your expanding business needs require
you to beef up the algorithm with user preferences or the like.
Note that "markup" in your content-to-translate generally leads to an
increased translation price tag.
> Consequences of A are that it is less work up-front,
> but more work for each language. And it also makes the
> modification of the localized pages challenging for
> developers not conversant in that language. Also, this
> solution does not address the output in log files, or
> other administrative aspects of the system, only that
> which is accessible through the web interface.
>
> Consequence of B are that it is more work up-front, but
> less work for each language thereafter. And it makes
> the single set of templates harder to read, and more
> complex as languages are added, but perhaps easier to
> maintain for language-challenged Yankee developers.
As you mentioned, internationalizing (I18N'ing) your application is a
non-trivial undertaking. Make sure you actually _must_ do it before
you go down this route; discuss the high costs involved (both in
development time and green backs) with management. I18N is _not_ the
place to skimp on an up front architecture investment. Like most
projects, if solid backing of a proper solution isn't available, you
probably want to steer clear of signing up for the project. :)
> I know that is what UTF-8 and Unicode are supposed to
> solve. Does it really work in the real world? Does
> that really handle all of the major languages?
In my experience, Unicode works in most cases. Java has excellent
support of Unicode -- it will statisfy enough of your needs that the
edge cases won't block you.
> As for sorting the data...
>
> This is an area about which I have found very little
> info and even less experience or exposure.
>
> This seems easier to describe with a scenario. Suppose
> our customer is a supplier in Europe who provides parts
> to manufacturers in various other countries. Their
> customers can view the site in their language of
> choice. And they enter data into the application in
> their own language as well. The data in the database
> is now in multiple languages.
>
> How do you handle the sorting of data in multiple
> languages? If you are using ORDER BY clauses in your
> SQL, how does the database know which language to use
> for sorting?
Your database must have support for multiple character sets. AFAIK,
stables releases of MySQL do not natively support more than one
language at a time. As a work around, you can trade in some of the
features of the RDBMS and store other languages in the same database
as binary.
Your JDBC driver must also behave in a reasonable fashion when given
data in alternate character sets. The MM JDBC driver must be told
explicitly not to do dumb character translations with data piped
through it via a query string argument of
"useUnicode=true&characterEncoding=ISO8859_1". We've made Mark
Matthews aware of the problems with his driver related to this, but he
isn't interested in correcting the behavior on the grounds that it
would degrade performance.
> Is it a common practice to include a column in the
> tables for a language code, so that you can SELECT only
> the records for a specific language?
>
> Or do people just avoid this problem and require their
> customers to enter data in a particular language?
I believe that we punt on this problem, assuming that each
installation of our application is used primarily by a user base who
all speak the same language.
> Or do developers just do the sorting in Java instead of
> in the database?
We do this in some cases.
> Or would you actually put the data into separate
> databases, or instances, one for each locale?
Yup.
> Can you tell that I have zero knowledge about this last
> aspect of the problem? 8^)
Depending upon your use case, you may be able to initially sidestep
this issue.
--
Daniel Rall <dl...@finemaltcoding.com>
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Yankee needs Localization advice
Posted by Daniel Rall <dl...@finemaltcoding.com>.
Eric Dobbs <er...@dobbse.net> writes:
> On Friday, August 30, 2002, at 11:40 AM, Eric Dobbs wrote:
>
> > 3. how the localized data gets sorted in the display
> > (or what the database must know about the language),
>
>
> On further investigation, I've found three methods of
> segregating language data in the database:
>
> 1) separate fields for each language
> (ID, NAME, DESCRIPTION, NAME_FR, DESCRIPTION_FR ...)
>
> 2) separate tables for each language
> (PRODUCT, PRODUCT_FR, PRODUCT_PL, ...)
>
> 3) a string localization table which is joined with the
> primary table
>
> select P.ID, L.NAME, L.DESCRIPTION
> from PRODUCT P, PRODUCT_STRING L
> where P.ID = L.FK_PRODUCT_ID
> and L.LANGUAGE = 'FR'
>
> Having found three methods for segregating data by
> locale, I'm lead to believe that this is the general
> practice. Can you confirm or deny that?
>
> Method 1 looks like The Wrong Way, especially if there
> are expected to be more than 2 languages.
>
> Method 2 looks like The Fast Way. It removes the
> performance hit of adding joins to every query
> involving localized strings.
>
> Method 3 looks like The Right Way, if you subscribe to
> the view that The Right Way is well normalized.
I concur, but point out that 3 is going to slow down your app. Still,
you couldn't pay me enough to be responsible for 1 or 2.
--
Daniel Rall <dl...@finemaltcoding.com>
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Yankee needs Localization advice
Posted by Eric Dobbs <er...@dobbse.net>.
On Friday, August 30, 2002, at 11:40 AM, Eric Dobbs wrote:
> 3. how the localized data gets sorted in the display
> (or what the database must know about the language),
On further investigation, I've found three methods of
segregating language data in the database:
1) separate fields for each language
(ID, NAME, DESCRIPTION, NAME_FR, DESCRIPTION_FR ...)
2) separate tables for each language
(PRODUCT, PRODUCT_FR, PRODUCT_PL, ...)
3) a string localization table which is joined with the
primary table
select P.ID, L.NAME, L.DESCRIPTION
from PRODUCT P, PRODUCT_STRING L
where P.ID = L.FK_PRODUCT_ID
and L.LANGUAGE = 'FR'
Having found three methods for segregating data by
locale, I'm lead to believe that this is the general
practice. Can you confirm or deny that?
Method 1 looks like The Wrong Way, especially if there
are expected to be more than 2 languages.
Method 2 looks like The Fast Way. It removes the
performance hit of adding joins to every query
involving localized strings.
Method 3 looks like The Right Way, if you subscribe to
the view that The Right Way is well normalized.
Anyone have experience with one or more of these
tricks? Any comments on how they work in the messy
world of real applications?
Thanks.
-Eric
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>