You are viewing a plain text version of this content. The canonical link for it is here.
Posted to modperl@perl.apache.org by Steve Hay <st...@uk.radan.com> on 2003/05/09 10:19:12 UTC
[OT] Character encodings in web pages
[Apologies if this question is slightly off-topic, but I don't really
know where else to post to.]
I have a problem reported by a German customer trying to view some web
pages (mod_perl output) containing characters with diacritical markings.
The page views OK except for all such special characters, which appear
garbled, e.g. a lowercase letter "a" with an umlaut (U+00E4) appears as
an uppercase letter "A" with a tilde (U+00C3) followed by a currency
sign (U+00A4).
The text concerned is read from a message file by the mod_perl content
generator. The web page that is it copied to is sent with the following
two HTTP headers (amongst others):
Content-Type: text/html; charset=ISO-8859-1
Content-Language: de-DE
and, for good measure, also contains the following two <META> elements:
<meta HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=ISO-8859-1">
<meta HTTP-EQUIV="Content-Language" CONTENT="de-DE">
(If I run the message file concerned through "od -c" then the lowercase
"a" umlaut is listed as "344".)
I can't reproduce the problem myself: I've tried viewing the web page
concerned in both NS7 and IE6, and the special characters all appear
correctly. (I have the whole setup (server & client) running on an
English Windows system; the customer's is running on a German Windows
system.)
Does anybody have any ideas what could be wrong, or suggestions of where
else to ask?
I wonder if the fact that characters are apparently all rendered as a
PAIR of characters, in which the first character is apparently always
uppercase "A" tilde, is significant?
Or is ISO-8859-1 simply the wrong character set for German?
Can any Germany speakers / HTTP gurus shed any light on this?
Thanks,
Steve
Re: [OT] Character encodings in web pages
Posted by Steve Hay <st...@uk.radan.com>.
Jim Morrison [Mailing-Lists] wrote:
>Steve,
>
>How do you get on with '£' signs? .. I have a very similar problem with
>A(umlaut)'s appearing in front of pound signs and various other chars.
>
I don't have any £ signs in these web pages. I tried inserting some
into my test program's text and XML files, but (of course) they display
perfectly for me...
>
>In my case it appears to be to do with LibXML.. You have to be very
>careful as I remember not to switch from UTF8 to iso-1859-1 or back to
>prevent them from appearing... I haven't had a chance to track down the
>latest incarnation of the bug but when I do I can give you a shout?
>
I'd be very grateful if you could tell me anything that you find out.
(And likewise, I'll report anything that I find in the meantime.)
As I say, I am using XML::LibXML too, so maybe that's it.
>
>All I can suggest for now is trying the perl-xml list? This problem
>seems to go about a bit?
>
Yes, I'll give that a try.
Thanks,
Steve
>
>Soz not to be more helpful :-)
>
>J.
>
>
>
>>-----Original Message-----
>>From: Steve Hay [mailto:steve.hay@uk.radan.com]
>>Sent: 09 May 2003 12:59
>>To: Dr. Helmut Zeilinger
>>Cc: modperl@perl.apache.org
>>Subject: Re: [OT] Character encodings in web pages
>>
>>
>>Dr. Helmut Zeilinger wrote:
>>
>>
>>
>>>My be it has nothing to do with your problem - some time
>>>
>>>
>>ago a work
>>
>>
>>>with XML::Parser and my page (erroniously) gave out some unicode
>>>characters (at least i think so). In that page, all the other
>>>"umlaute" where not shown correctly by the browser (IE 6).
>>>
>>>
>>This sounds like it could be relevant: the page being generated does
>>contain data from XML files too (parsed with XML::LibXML), and you're
>>the second person on this list to make this suggestion, so I tried
>>adding some XML parsed content to the page output by my test program
>>(new version attached).
>>
>>Alas, all my a-umlauts still display correctly: one from an XML
>>attribute, one from XML text content, and one from the plain text
>>message file. No doubt it'll run fine for you too.
>>
>>However, some of the content on the page being generated is
>>user input
>>from elsewhere in the application (CGI input from previous
>>pages, data
>>read from plain text files and data parsed from XML files),
>>so it may be
>>the case that my customer's input includes some characters
>>that make the
>>page "break", but when I try it it "works" simply because I have
>>different input that doesn't include any such characters.
>>
>>I will look into this more closely.
>>
>>If anybody knows of, or finds any, particular characters that
>>cause the
>>a-umlauts to be shown incorrectly I'd be interested to hear.
>>
>>Steve
>>
>>
>>
>>
>
>
>
>
>
RE: [OT] Character encodings in web pages
Posted by Mithun Bhattacharya <in...@yahoo.com>.
--- "Jim Morrison [Mailing-Lists]" <ju...@mediaisotope.com> wrote:
> Steve,
>
> How do you get on with '�' signs? .. I have a very similar problem
> with
> A(umlaut)'s appearing in front of pound signs and various other
> chars.
>
> In my case it appears to be to do with LibXML.. You have to be very
> careful as I remember not to switch from UTF8 to iso-1859-1 or back
> to
In my case the XML file was modifiable by the users therefore I kept
the XML in ISO-8859-1 before parsing I converted it to UTF-8 using
Unicode::String and post parsing any text which has to go to the HTML I
convert back to ISO-8859-1 using Unicode::String. I have the luxury of
assumming that the user of my system is going to use ISO-8859-1 This
might not apply to others.
Mithun
__________________________________
Do you Yahoo!?
The New Yahoo! Search - Faster. Easier. Bingo.
http://search.yahoo.com
RE: [OT] Character encodings in web pages
Posted by "Jim Morrison [Mailing-Lists]" <ju...@mediaisotope.com>.
Steve,
How do you get on with '£' signs? .. I have a very similar problem with
A(umlaut)'s appearing in front of pound signs and various other chars.
In my case it appears to be to do with LibXML.. You have to be very
careful as I remember not to switch from UTF8 to iso-1859-1 or back to
prevent them from appearing... I haven't had a chance to track down the
latest incarnation of the bug but when I do I can give you a shout?
All I can suggest for now is trying the perl-xml list? This problem
seems to go about a bit?
Soz not to be more helpful :-)
J.
> -----Original Message-----
> From: Steve Hay [mailto:steve.hay@uk.radan.com]
> Sent: 09 May 2003 12:59
> To: Dr. Helmut Zeilinger
> Cc: modperl@perl.apache.org
> Subject: Re: [OT] Character encodings in web pages
>
>
> Dr. Helmut Zeilinger wrote:
>
> > My be it has nothing to do with your problem - some time
> ago a work
> > with XML::Parser and my page (erroniously) gave out some unicode
> > characters (at least i think so). In that page, all the other
> > "umlaute" where not shown correctly by the browser (IE 6).
>
> This sounds like it could be relevant: the page being generated does
> contain data from XML files too (parsed with XML::LibXML), and you're
> the second person on this list to make this suggestion, so I tried
> adding some XML parsed content to the page output by my test program
> (new version attached).
>
> Alas, all my a-umlauts still display correctly: one from an XML
> attribute, one from XML text content, and one from the plain text
> message file. No doubt it'll run fine for you too.
>
> However, some of the content on the page being generated is
> user input
> from elsewhere in the application (CGI input from previous
> pages, data
> read from plain text files and data parsed from XML files),
> so it may be
> the case that my customer's input includes some characters
> that make the
> page "break", but when I try it it "works" simply because I have
> different input that doesn't include any such characters.
>
> I will look into this more closely.
>
> If anybody knows of, or finds any, particular characters that
> cause the
> a-umlauts to be shown incorrectly I'd be interested to hear.
>
> Steve
>
>
Re: [OT] Character encodings in web pages
Posted by Steve Hay <st...@uk.radan.com>.
Dr. Helmut Zeilinger wrote:
> My be it has nothing to do with your problem - some time ago a work with
> XML::Parser and my page (erroniously) gave out some unicode characters
> (at least i think so).
> In that page, all the other "umlaute" where not shown correctly by the
> browser (IE 6).
This sounds like it could be relevant: the page being generated does
contain data from XML files too (parsed with XML::LibXML), and you're
the second person on this list to make this suggestion, so I tried
adding some XML parsed content to the page output by my test program
(new version attached).
Alas, all my a-umlauts still display correctly: one from an XML
attribute, one from XML text content, and one from the plain text
message file. No doubt it'll run fine for you too.
However, some of the content on the page being generated is user input
from elsewhere in the application (CGI input from previous pages, data
read from plain text files and data parsed from XML files), so it may be
the case that my customer's input includes some characters that make the
page "break", but when I try it it "works" simply because I have
different input that doesn't include any such characters.
I will look into this more closely.
If anybody knows of, or finds any, particular characters that cause the
a-umlauts to be shown incorrectly I'd be interested to hear.
Steve
Re: [OT] Character encodings in web pages
Posted by "Dr. Helmut Zeilinger" <h....@envirotex.de>.
I let it run under ModPerl::Registry (so its IT again).
I even added an uppercase "Ä".
NS 7 (Linux), IE 6 and wget showed (got) the correct output - (sorry)
My be it has nothing to do with your problem - some time ago a work with
XML::Parser and my page (erroniously) gave out some unicode characters (at
least i think so).
In that page, all the other "umlaute" where not shown correctly by the
browser (IE 6).
Helmut
--On Friday, May 09, 2003 10:57:29 +0100 Steve Hay <st...@uk.radan.com>
wrote:
> Dr. Helmut Zeilinger wrote:
>
>> Hi,
>>
>> could you give us a link to have a look on this site?
>
> I wish I could, but its an intranet site :-(
>
> I've attached a msg.tar.gz which contains two files: a CGI script (not
> mod_perl -- I did say this was slightly OT ;-) and a sample message file.
> If I drop these into my cgi-bin folder and browse to /cgi-bin/msg.pl then
> the German message that I see is shown correctly, but the customer sees
> the a-umlaut corrupted. Does it work for you?
>
>>
>>
>> Did you try for example chr(228) for a lowercase a-umlaut (ä) and
>> chr(196) for an uppercase A-Umlaut (Ä)? This works for our site.
>
> I am using chr(228) -- the output of "od -c" is octal (od == "Octal
> Dump"), and octal 344 is decimal 228.
>
> Double-check: running
>
> perl -e "print chr(228)" | od -c
>
> outputs
>
> 0000000 344
> 0000001
>
>>
>> As you, we have
>>
>> <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
>
> Well that's encouraging, at least.
>
>
>
Re: [OT] Character encodings in web pages
Posted by Steve Hay <st...@uk.radan.com>.
Dr. Helmut Zeilinger wrote:
> Hi,
>
> could you give us a link to have a look on this site?
I wish I could, but its an intranet site :-(
I've attached a msg.tar.gz which contains two files: a CGI script (not
mod_perl -- I did say this was slightly OT ;-) and a sample message
file. If I drop these into my cgi-bin folder and browse to
/cgi-bin/msg.pl then the German message that I see is shown correctly,
but the customer sees the a-umlaut corrupted. Does it work for you?
>
>
> Did you try for example chr(228) for a lowercase a-umlaut (ä) and
> chr(196) for an uppercase A-Umlaut (Ä)? This works for our site.
I am using chr(228) -- the output of "od -c" is octal (od == "Octal
Dump"), and octal 344 is decimal 228.
Double-check: running
perl -e "print chr(228)" | od -c
outputs
0000000 344
0000001
>
> As you, we have
>
> <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
Well that's encouraging, at least.
Re: [OT] Character encodings in web pages
Posted by "Dr. Helmut Zeilinger" <h....@envirotex.de>.
Hi,
could you give us a link to have a look on this site?
Did you try for example chr(228) for a lowercase a-umlaut (ä) and
chr(196) for an uppercase A-Umlaut (Ä)? This works for our site.
As you, we have
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
Helmut
--On Friday, May 09, 2003 09:19:12 +0100 Steve Hay <st...@uk.radan.com>
wrote:
> [Apologies if this question is slightly off-topic, but I don't really
> know where else to post to.]
>
> I have a problem reported by a German customer trying to view some web
> pages (mod_perl output) containing characters with diacritical markings.
> The page views OK except for all such special characters, which appear
> garbled, e.g. a lowercase letter "a" with an umlaut (U+00E4) appears as
> an uppercase letter "A" with a tilde (U+00C3) followed by a currency sign
> (U+00A4).
>
> The text concerned is read from a message file by the mod_perl content
> generator. The web page that is it copied to is sent with the following
> two HTTP headers (amongst others):
>
> Content-Type: text/html; charset=ISO-8859-1
> Content-Language: de-DE
>
> and, for good measure, also contains the following two <META> elements:
>
> <meta HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=ISO-8859-1">
> <meta HTTP-EQUIV="Content-Language" CONTENT="de-DE">
>
> (If I run the message file concerned through "od -c" then the lowercase
> "a" umlaut is listed as "344".)
>
> I can't reproduce the problem myself: I've tried viewing the web page
> concerned in both NS7 and IE6, and the special characters all appear
> correctly. (I have the whole setup (server & client) running on an
> English Windows system; the customer's is running on a German Windows
> system.)
>
> Does anybody have any ideas what could be wrong, or suggestions of where
> else to ask?
>
> I wonder if the fact that characters are apparently all rendered as a
> PAIR of characters, in which the first character is apparently always
> uppercase "A" tilde, is significant?
>
> Or is ISO-8859-1 simply the wrong character set for German?
>
> Can any Germany speakers / HTTP gurus shed any light on this?
>
> Thanks,
>
> Steve
>
>
Re: [OT] Character encodings in web pages
Posted by Martin Langhoff <ml...@nzl.com.ar>.
Steve,
your user is using IE6 -- which should default to the correct charset.
Maybe your client is forcing it? This is straightforward to do: normally
'auto' is selected in the relevant menu, but he could have set it to
something else and forgotten.
Either that, or a broken proxy may be involved that messes up the content.
Has your client got other browsers/machines to test with? Or can the
client use a CLI utility such as wget to check he is actually getting
the same file you are? (ask him to wget the file and send it over to you)
regards,
martin
Re: [OT] Character encodings in web pages
Posted by Mithun Bhattacharya <in...@yahoo.com>.
I have noticed something simillar when I ended up mixing UTF-8 encoded
characters in a HTML with Content-Encoding set to ISO-8859-1 The
problem used to occur only for the 8bit characters ofcourse.
Possible situation could be if the message file has been parsed using
some parser. Most parsers especially XML based will give only UTF-8
encoded characters.
Mithun
--- Steve Hay <st...@uk.radan.com> wrote:
> [Apologies if this question is slightly off-topic, but I don't really
>
> know where else to post to.]
>
> I have a problem reported by a German customer trying to view some
> web
> pages (mod_perl output) containing characters with diacritical
> markings.
> The page views OK except for all such special characters, which
> appear
> garbled, e.g. a lowercase letter "a" with an umlaut (U+00E4) appears
> as
> an uppercase letter "A" with a tilde (U+00C3) followed by a currency
> sign (U+00A4).
>
> The text concerned is read from a message file by the mod_perl
> content
> generator. The web page that is it copied to is sent with the
__________________________________
Do you Yahoo!?
The New Yahoo! Search - Faster. Easier. Bingo.
http://search.yahoo.com
Re: [OT] Character encodings in web pages
Posted by Steve Hay <st...@uk.radan.com>.
Ged Haywood wrote:
>Hi Steve,
>
>On Fri, 9 May 2003, Steve Hay wrote:
>
>
>
>> The page views OK except for all such special characters, which appear
>>garbled, e.g. a lowercase letter "a" with an umlaut (U+00E4) appears as
>>an uppercase letter "A" with a tilde (U+00C3) followed by a currency
>>sign (U+00A4).
>>
>>
>
>I see things like that with NS4.7 - is that what your customer is using?
>
No - he's using IE6.
Re: [OT] Character encodings in web pages
Posted by Ged Haywood <ge...@www2.jubileegroup.co.uk>.
Hi Steve,
On Fri, 9 May 2003, Steve Hay wrote:
> The page views OK except for all such special characters, which appear
> garbled, e.g. a lowercase letter "a" with an umlaut (U+00E4) appears as
> an uppercase letter "A" with a tilde (U+00C3) followed by a currency
> sign (U+00A4).
I see things like that with NS4.7 - is that what your customer is using?
Reply off-list if you like.
73,
Ged.