You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@shindig.apache.org by Justin Wyllie <ju...@hotmail.co.uk> on 2010/07/20 13:18:10 UTC
Bug in PHP Shindig: non UTF-8 gadgets lose all non asci characters
The original problem which I posted to the users list was that gadgets with non UTF-8 encodings (I used iso-8859-1 to test) were losing all non ascii characters in both the title (metadata call) and content (gadget rendering call).
Details of the problem and solution is as follows:
In BasicRemoteContentFetcher this line:
$content = mb_convert_encoding($content, 'UTF-8', $charset);
converts the fetched XML as a string to UTF-8 whatever encoding it was in. ($charset is the source encoding)
But the xml declaration line was not touched. So, after this we may have a gadget like this:
<?xml version="1.0" encoding="iso-8859-1"?><Module> <ModulePrefs title="IñtërnâtiônàlizætiønX" /> <Content type="html"> <![CDATA[ ]]> </Content> </Module>
which is UTF-8 encoded but with an iso-8859-1 encoding attribute.
Later in the call (metadata request or gadget rendering) in GadgetSpecParser->parse() we load the XML content into an XML DOM object. At this point the error occurs - naturally as the UTF-8 content is flagged as being in iso-8859-1.
My fix is as follows:
In BasicRemoteContentFetcher->parseResult replace:
$content = mb_convert_encoding($content, 'UTF-8', $charset);
with
$content = mb_convert_encoding($content, 'UTF-8', $charset); $pattern = 'encoding=\s*([' . '\'"])' . $charset . '\s*\1'; $content = mb_ereg_replace($pattern,'encoding="UTF-8"',$content,"i") ;
Now the XML is UTF-8 encoded and has the correct UTF-8 encoding attribute.
Justin
_________________________________________________________________
http://clk.atdmt.com/UKM/go/197222280/direct/01/
Do you have a story that started on Hotmail? Tell us now
Re: Bug in PHP Shindig: non UTF-8 gadgets lose all non asci characters
Posted by Bastian Hofmann <ba...@googlemail.com>.
Hi,
I tested this and could verify this behavior. I created a patch which
fixes this issue as well as another issue which occurs, when the
content type is not send via an http header but is only present in the
XML declaration.
Please review this at http://codereview.appspot.com/1952044/
Cheers
Bastian
2010/8/18 Paul Lindner <pl...@linkedin.com>:
> I'm not sure if you read this. Do you think it's a correct assessment?
> ---------- Forwarded message ----------
> From: Justin Wyllie <ju...@hotmail.co.uk>
> Date: Tue, Jul 20, 2010 at 4:18 AM
> Subject: Bug in PHP Shindig: non UTF-8 gadgets lose all non asci characters
> To: dev@shindig.apache.org
>
>
>
> The original problem which I posted to the users list was that gadgets with
> non UTF-8 encodings (I used iso-8859-1 to test) were losing all non ascii
> characters in both the title (metadata call) and content (gadget rendering
> call).
> Details of the problem and solution is as follows:
>
> In BasicRemoteContentFetcher this line:
> $content = mb_convert_encoding($content, 'UTF-8', $charset);
> converts the fetched XML as a string to UTF-8 whatever encoding it was in.
> ($charset is the source encoding)
> But the xml declaration line was not touched. So, after this we may have a
> gadget like this:
> <?xml version="1.0" encoding="iso-8859-1"?><Module> <ModulePrefs
> title="IñtërnâtiônàlizætiønX" /> <Content type="html"> <![CDATA[
> ]]> </Content> </Module>
> which is UTF-8 encoded but with an iso-8859-1 encoding attribute.
> Later in the call (metadata request or gadget rendering) in
> GadgetSpecParser->parse() we load the XML content into an XML DOM object. At
> this point the error occurs - naturally as the UTF-8 content is flagged as
> being in iso-8859-1.
> My fix is as follows:
> In BasicRemoteContentFetcher->parseResult replace:
> $content = mb_convert_encoding($content, 'UTF-8', $charset);
> with
> $content = mb_convert_encoding($content, 'UTF-8', $charset); $pattern =
> 'encoding=\s*([' . '\'"])' . $charset . '\s*\1'; $content =
> mb_ereg_replace($pattern,'encoding="UTF-8"',$content,"i") ;
> Now the XML is UTF-8 encoded and has the correct UTF-8 encoding attribute.
> Justin
>
>
>
>
>
>
>
> _________________________________________________________________
> http://clk.atdmt.com/UKM/go/197222280/direct/01/
> Do you have a story that started on Hotmail? Tell us now
>
>
> --
> Paul Lindner -- plindner@linkedin.com -- linkedin.com/in/plindner
>