You are viewing a plain text version of this content. The canonical link for it is here.
Posted to modperl@perl.apache.org by angie ahl <an...@gmail.com> on 2005/02/11 00:51:28 UTC

[mp2] utf8 data garbled on second + subsequent loads

Firstly may I appologise for the length of this, I fear I won't get to
the bottom of it otherwise.

All of the following is being done on OS X 10.3
Server: Apache/2.0.52 (Unix) mod_perl/1.999.21 Perl/v5.8.6

I noticed today that some of my utf8 data was becoming corrupted but
couldn't see why.
Hebrew and Arabic were fine but characters like ü were becoming garbled.

So I went back to a basic test script I wrote for testing utf8 and
this was fine running under cgi mode.

So I converted the cgi script to a package. 

Wrapped the main code in sub handler {} and added return Apache::OK

both the cgi script and package have:

use strict;
use warnings;
use 5.008006;

use utf8;
use DBI;
use CGI (':standard');
use Encode qw/is_utf8 decode/;

binmode(STDOUT, ":utf8");

When I load the cgi script everything is fine every time.

When I load the package it's fine the first time but becomes garbled
on susbsequent loads.

Restarting apache makes the first load fine again. Quitting the
browser and relaunching it also solves the problem.

Here's the httpd.conf info:

   PerlModule ModPerl::Registry

<FilesMatch "\.html$">
	SetHandler perl-script 
	Perlhandler unidbtest
</FilesMatch>

I'm not sure if ModPerl::Registry is involved or not. I've tried both
perl-script and modperl as the handler.

First load I get like this as the output (whole page so you can see
the meta tags etc):

<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html
	PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
	 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en-US"
xml:lang="en-US"><head><title>Simple UTF 8 test</title>
</head><body>


<form method="get" action="/test.html"
enctype="application/x-www-form-urlencoded">
<h1>Unicode test: Page 1</h1><table border="1"
cellpadding="5"><tr><td>description</td><td>language</td><td>char in
unicode</td></td>
<tr><td>abimer</td><td>french</td><td><input type="text" name="abimer"
value="abîmer" size="50" maxlength="80" /></td></td>
<tr><td>angies</td><td>mixed</td><td><input type="text" name="angies"
value="łç☺Ą☺Ąreçenu☺" size="50" maxlength="80" /></td></td>

<tr><td>aogonec</td><td>polish</td><td><input type="text"
name="aogonec" value="Ą" size="50" maxlength="80" /></td></td>
<tr><td>citroen</td><td>french</td><td><input type="text"
name="citroen" value="citroën" size="50" maxlength="80" /></td></td>
<tr><td>disco</td><td>french</td><td><input type="text" name="disco"
value="discothèque" size="50" maxlength="80" /></td></td>
<tr><td>hebrew_alef</td><td>hebrew</td><td><input type="text"
name="hebrew_alef" value="א" size="50" maxlength="80" /></td></td>
<tr><td>lslash</td><td>polish</td><td><input type="text" name="lslash"
value="ł" size="50" maxlength="80" /></td></td>
<tr><td>recenu</td><td>french</td><td><input type="text" name="recenu"
value="reçenu" size="50" maxlength="80" /></td></td>

<tr><td>smiley</td><td>none</td><td><input type="text" name="smiley"
value="☺" size="50" maxlength="80" /></td></td>
<tr><td>zcaron</td><td>czech</td><td><input type="text" name="zcaron"
value="Ž" size="50" maxlength="80" /></td></td>
</table><input type="hidden" name="VertDo" value="test unicode" 
/><input type="submit" name="Do" value="test unicode"
/><div></div></form>

</body></html>

on subsequent loads the following happens:

abîmer  becomes ab�mer
citroën  becomes citro�n

and so on.

If I leave my machine alone for a while and then load the page I get
correct output again, subsequent loads are garbled again.

really very confused about what's going on.

What to do next? I'm happy to put both scripts somewhere for download
if anyone wants to replicate the problem.

Thanks

Angie

Re: [mp2] utf8 data garbled on second + subsequent loads

Posted by angie ahl <an...@gmail.com>.
Thanks

I'll give that a go. What OS are you on out of interest. As I just
decided to try in on one of my Linux servers and I'm not having that
problem there at all. So I'm now wondering if its a bug in a specific
version or maybe even OS X.

Thanks

Angie


On Fri, 11 Feb 2005 13:56:55 +0100, Roman Vašíček <rv...@petamem.com> wrote:
> It is strange. CGI works ok, but i was unable to get correct output even
> from module even first time. It seems to be ok for me after switching
> from perl-script to modperl handler.
> 
> apache          2.0.48
> mod_perl        1.99_14
> perl            5.0.8
> 
> <Location /unitest>
>         SetHandler modperl
> # perl-script does not work - don't know why
> #        SetHandler perl-script
>         PerlResponseHandler Unitest
> </Location>
> 
> Here is changed handler and startpage code
> 
> sub handler {
>   my $r = shift;
>   $r->print( header(-type => "text/html",  -charset => "utf-8"));
>   $r->print( start_html(
>                         # This seems to be no important?
>                         -encoding => 'utf-8',
>                         -title => "Simple UTF 8 test"));
>   $r->print( "\n\n");
>   # page calls
>   unless (param('VertDo')) {
>     $r->print(&startpage());
>   }
>   $r->print( "\n\n", end_html);
>   return Apache::OK;
>   #1;
> }
> 
> sub startpage {
>   my %uni = &unihash;
>   my $p = '';
>   $p = "\n";
>   $p .= start_form(-method=>"get", -action=>"/test.html");
>   $p .= h1("Unicode test: Page 1");
>   $p .= '<table border="1" cellpadding="5">';
>   my $tablinedef = "<tr>" . "<td>%s</td>" x 3 . "</tr>\n";
>   # Headerline
>   $p .= sprintf $tablinedef,
>     "description", "language", "char in unicode";
>   foreach (sort keys %uni) {
>     $p .= sprintf $tablinedef,
>       $_ ,
>         $uni{$_}->{language},
>           textfield(-name=>$_,
>                     -default=>$uni{$_}->{character}, -size=>50, -maxlength=>80);
>   }
>   $p .= '</table>';
>   $p .= hidden(-name=>'VertDo',
>                -default=>['test unicode']);
>   $p .= submit(-name=>'Do',
>                -value=>'test unicode');
>   $p .= endform;
>   return $p;
> }
> 
> Hope it helps
>   Roman
> 
> On Fri, Feb 11, 2005 at 11:47:49AM +0000, angie ahl wrote:
> > I've looked into it further and stripped it right back to the start.
> >
> > Now I just have the unicode chars hard coded in the script and
> > printing them to a table.
> >
> > I'm not even passing them through a form now, just printing them.
> >
> > In CGI mode it works perfectly and in MP2 mode it loads fine the first
> > time after apache or the browser is restarted. Then goes kaboom, ie
> > gibberish
> >
> > Here's both scripts.
> [...]
> 
> --
>  best regards
>   Ing. Roman Vasicek
> 
>  software developer
> +----------------------------------------------------------------------------+
>  PetaMem s.r.o., Ocelarska 1, 190 00 Praha 9 - Liben, Czech Republic
>  http://www.petamem.com/
>

Re: [mp2] utf8 data garbled on second + subsequent loads

Posted by Stas Bekman <st...@stason.org>.
Roman Vašíček wrote:
> On Fri, Feb 11, 2005 at 01:56:55PM +0100, Roman Vašíček wrote:
> 
>>It is strange. CGI works ok, but i was unable to get correct output even 
>>from module even first time. It seems to be ok for me after switching 
>>from perl-script to modperl handler.
>>
>>apache		2.0.48
>>mod_perl	1.99_14
>>perl		5.0.8
> 
> 
> Perl version is 5.8.0 of course :-) OS is SuSE Linux. Apache and perl are 
> from distro, mod_perl is self-compiled.

It's not clear whether you've resolved this issue or not. If not please 
post here a very short handler/script that we can reliably reproduce the 
problem and we will look at it. thanks.


-- 
__________________________________________________________________
Stas Bekman            JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/     mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org   http://ticketmaster.com

Re: [mp2] utf8 data garbled on second + subsequent loads

Posted by Roman Vašíček <rv...@petamem.com>.
On Fri, Feb 11, 2005 at 01:56:55PM +0100, Roman Vašíček wrote:
> It is strange. CGI works ok, but i was unable to get correct output even 
> from module even first time. It seems to be ok for me after switching 
> from perl-script to modperl handler.
> 
> apache		2.0.48
> mod_perl	1.99_14
> perl		5.0.8

Perl version is 5.8.0 of course :-) OS is SuSE Linux. Apache and perl are 
from distro, mod_perl is self-compiled.

-- 
 best regards
  Ing. Roman Vasicek

 software developer
+----------------------------------------------------------------------------+
 PetaMem s.r.o., Ocelarska 1, 190 00 Praha 9 - Liben, Czech Republic
 http://www.petamem.com/

Re: [mp2] utf8 data garbled on second + subsequent loads

Posted by Roman Vašíček <rv...@petamem.com>.
It is strange. CGI works ok, but i was unable to get correct output even 
from module even first time. It seems to be ok for me after switching 
from perl-script to modperl handler.

apache		2.0.48
mod_perl	1.99_14
perl		5.0.8

<Location /unitest>
        SetHandler modperl
# perl-script does not work - don't know why
#        SetHandler perl-script
        PerlResponseHandler Unitest
</Location>

Here is changed handler and startpage code

sub handler {
  my $r = shift;
  $r->print( header(-type => "text/html",  -charset => "utf-8"));
  $r->print( start_html(
                        # This seems to be no important?
                        -encoding => 'utf-8',
                        -title => "Simple UTF 8 test"));
  $r->print( "\n\n");
  # page calls
  unless (param('VertDo')) {
    $r->print(&startpage());
  }
  $r->print( "\n\n", end_html);
  return Apache::OK;
  #1;
}

sub startpage {
  my %uni = &unihash;
  my $p = '';
  $p = "\n";
  $p .= start_form(-method=>"get", -action=>"/test.html");
  $p .= h1("Unicode test: Page 1");
  $p .= '<table border="1" cellpadding="5">';
  my $tablinedef = "<tr>" . "<td>%s</td>" x 3 . "</tr>\n";
  # Headerline
  $p .= sprintf $tablinedef,
    "description", "language", "char in unicode";
  foreach (sort keys %uni) {
    $p .= sprintf $tablinedef,
      $_ ,
        $uni{$_}->{language},
          textfield(-name=>$_,
                    -default=>$uni{$_}->{character}, -size=>50, -maxlength=>80);
  }
  $p .= '</table>';
  $p .= hidden(-name=>'VertDo',
               -default=>['test unicode']);
  $p .= submit(-name=>'Do',
               -value=>'test unicode');
  $p .= endform;
  return $p;
}

Hope it helps
  Roman

On Fri, Feb 11, 2005 at 11:47:49AM +0000, angie ahl wrote:
> I've looked into it further and stripped it right back to the start.
> 
> Now I just have the unicode chars hard coded in the script and
> printing them to a table.
> 
> I'm not even passing them through a form now, just printing them.
> 
> In CGI mode it works perfectly and in MP2 mode it loads fine the first
> time after apache or the browser is restarted. Then goes kaboom, ie
> gibberish
> 
> Here's both scripts.
[...]

-- 
 best regards
  Ing. Roman Vasicek

 software developer
+----------------------------------------------------------------------------+
 PetaMem s.r.o., Ocelarska 1, 190 00 Praha 9 - Liben, Czech Republic
 http://www.petamem.com/

Re: [mp2] utf8 data garbled on second + subsequent loads

Posted by angie ahl <an...@gmail.com>.
I've looked into it further and stripped it right back to the start.

Now I just have the unicode chars hard coded in the script and
printing them to a table.

I'm not even passing them through a form now, just printing them.

In CGI mode it works perfectly and in MP2 mode it loads fine the first
time after apache or the browser is restarted. Then goes kaboom, ie
gibberish

Here's both scripts.

_______ START CGI SCRIPT _______

#!/usr/local/bin/perl
#
#       Author Angie Ahl (angie@vertebrate.co.uk) : 2004/12/01 22:14:00
#
# script to test unicode, standard cgi mode.
# requires perl 5.8, DBI, DBD::MySQL, CGI and Encode

use strict;
use warnings;
use 5.008006;

use utf8;
use DBI;
use CGI (':standard');
use Encode qw/is_utf8 decode/;

binmode(STDOUT, ":utf8");

                        print header(-type => "text/html",  -charset
=> "utf-8");
                        print start_html(
                                        # This seems to be no important?
                                         -encoding => 'utf-8',
                                        -title => "Simple UTF 8 test");
                        print "\n\n";

# page calls
        unless (param('VertDo')) {
                        &startpage;
        }

        print "\n\n", end_html;

sub startpage {
        my %uni = &unihash;

        print "\n";
        print start_form(-method=>"get", -action=>"/test.html");
        print h1("Unicode test: Page 1");
        print '<table border="1" cellpadding="5">';

        my $tablinedef = "<tr>" . "<td>%s</td>" x 3 . "</td>\n";
        # Headerline
        printf $tablinedef,
                        "description","language","char in unicode";

        foreach (sort keys %uni) {
                        printf $tablinedef,
                        $_,
                        $uni{$_}->{language},
                        textfield(-name=>$_,
-default=>$uni{$_}->{character}, -size=>50, -maxlength=>80);
        }
        print '</table>';
        print hidden(-name=>'VertDo',
                             -default=>['test unicode']);

        print submit(-name=>'Do',
                        -value=>'test unicode');
        print endform;

}

# subroutines

sub unihash {
        my %uni = (
                        hebrew_alef => {
                                        character => chr(0x05d0),
                                        language => "hebrew",
                        },
                        smiley => {
                                        character => "\x{263a}",
                                        language => "none",
                        },
                        zcaron => {
                                        character => "\x{017d}",
                                        language => "czech",
                        },
                        aogonec => {
                                        character => "\x{0104}",
                                        language => "polish",
                        },
                        lslash => {
                                        character => "\x{0142}",
                                        language => "polish",
                        },
                        recenu => {
                                        character => "re\x{e7}enu",
                                        language => "french",
                        },
                        citroen => {
                                        character => "citro\x{eb}n",
                                        language => "french",
                        },
                        abimer => {
                                        character => "ab\x{ee}mer",
                                        language => "french",
                        },
                        disco => {
                                        character => "discoth\x{e8}que",
                                        language => "french",
                        },
                        angies => {
                                        character =>
"\x{0142}\x{e7}\x{263a}\x{0104}\x{263a}\x{0104}re\x{e7}enu\x{263a}",
                                        language => "mixed",
                        },
        );
return %uni;
}

_______ END CGI SCRIPT _______

_______ START MP2 PACKAGE _______

package unitest;

#!/usr/local/bin/perl
#
#       Author Angie Ahl (angie@vertebrate.co.uk) : 2004/12/01 22:14:00
#
# script to test unicode, standard cgi mode.
# requires perl 5.8, DBI, DBD::MySQL, CGI and Encode

use strict;
use warnings;
use 5.008006;

use utf8;
use DBI;
use CGI (':standard');
use Encode qw/is_utf8 decode/;

binmode(STDOUT, ":utf8");

sub handler {

                        print header(-type => "text/html",  -charset
=> "utf-8");
                        print start_html(
                                        # This seems to be no important?
                                         -encoding => 'utf-8',
                                        -title => "Simple UTF 8 test");
                        print "\n\n";

# page calls
        unless (param('VertDo')) {
                        &startpage;
        }

        print "\n\n", end_html;

        return Apache::OK;
        #1;
}

sub startpage {
        my %uni = &unihash;

        print "\n";
        print start_form(-method=>"get", -action=>"/test.html");
        print h1("Unicode test: Page 1");
        print '<table border="1" cellpadding="5">';

        my $tablinedef = "<tr>" . "<td>%s</td>" x 3 . "</td>\n";
        # Headerline
        printf $tablinedef,
                        "description","language","char in unicode";

        foreach (sort keys %uni) {
                        printf $tablinedef,
                        $_,
                        $uni{$_}->{language},
                        textfield(-name=>$_,
-default=>$uni{$_}->{character}, -size=>50, -maxlength=>80);
        }
        print '</table>';
        print hidden(-name=>'VertDo',
                             -default=>['test unicode']);

        print submit(-name=>'Do',
                        -value=>'test unicode');
        print endform;

}

# subroutines

sub unihash {
        my %uni = (
                        hebrew_alef => {
                                        character => chr(0x05d0),
                                        language => "hebrew",
                        },
                        smiley => {
                                        character => "\x{263a}",
                                        language => "none",
                        },
                        zcaron => {
                                        character => "\x{017d}",
                                        language => "czech",
                        },
                        aogonec => {
                                        character => "\x{0104}",
                                        language => "polish",
                        },
                        lslash => {
                                        character => "\x{0142}",
                                        language => "polish",
                        },
                        recenu => {
                                        character => "re\x{e7}enu",
                                        language => "french",
                        },
                        citroen => {
                                        character => "citro\x{eb}n",
                                        language => "french",
                        },
                        abimer => {
                                        character => "ab\x{ee}mer",
                                        language => "french",
                        },
                        disco => {
                                        character => "discoth\x{e8}que",
                                        language => "french",
                        },
                        angies => {
                                        character =>
"\x{0142}\x{e7}\x{263a}\x{0104}\x{263a}\x{0104}re\x{e7}enu\x{263a}",
                                        language => "mixed",
                        },
        );
return %uni;
}

_______ END MP2 PACKAGE _______

Re: [mp2] utf8 data garbled on second + subsequent loads

Posted by Roman Vašíček <rv...@petamem.com>.
I expect multiple decoding of the same text. Try to use conditional decoding

	$text = Encode::decode_utf8($text) if !Encode::is_utf8($text);

or let us look at your code. If you are sure that source data encoding is
utf8 and only utf8 flag is missing it is possible to use (but probably not 
suggested)

	Encode::_utf8_on($text)

  Roman


On Thu, Feb 10, 2005 at 11:51:28PM +0000, angie ahl wrote:
> Firstly may I appologise for the length of this, I fear I won't get to
> the bottom of it otherwise.
> 
> All of the following is being done on OS X 10.3
> Server: Apache/2.0.52 (Unix) mod_perl/1.999.21 Perl/v5.8.6
> 
> I noticed today that some of my utf8 data was becoming corrupted but
> couldn't see why.
> Hebrew and Arabic were fine but characters like ü were becoming garbled.
[...]

-- 
 best regards
  Ing. Roman Vasicek

 software developer
+----------------------------------------------------------------------------+
 PetaMem s.r.o., Ocelarska 1, 190 00 Praha 9 - Liben, Czech Republic
 http://www.petamem.com/