You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Vasja Ocvirk <va...@vizija.si> on 2007/09/11 11:58:49 UTC

UTF-16 problem

Does anyone know what to do if Nutch doesn't crawl and index web pages 
in UTF-16? Did anyone had such a problem yet?

Best regards,
Vasja

Re: UTF-16 problem

Posted by Vasja Ocvirk <va...@vizija.si>.
The problem is in charset. If we change charset of the page from UTF-16 
to UTF-8 then Nutch fetches all urls on that page. If a page has charset 
UTF-16 then Nutch fetches just the first url but not the urls on that 
page like with the utf8 charset. Here are two examples:

UTF16 - no additional urls for next cycle
<html>
<head>
   <meta http-equiv="Content-Type" content="text/html; charset=utf-16" />
</head>
<body>
   <a href="slo.php">sdfsdf</a>
   <a class="ai" href="info.aspx?docid=54046">test</a>
   <a href="http://www.something.com">something.com</a>
</body>
</html>

UTF8 - first two urls are fetched for next cycle and this is OK.
<html>
<head>
   <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
</head>
<body>
   <a href="slo.php">sdfsdf</a>
   <a class="ai" href="info.aspx?docid=54046">test</a>
   <a href="http://www.something.com">something.com</a>
</body>
</html>

Thanks!

Best regards,
Vasja

Doğacan Güney wrote:
> On 9/11/07, Vasja Ocvirk <va...@vizija.si> wrote:
>   
>> Does anyone know what to do if Nutch doesn't crawl and index web pages
>> in UTF-16? Did anyone had such a problem yet?
>>     
>
> Nutch should work with UTF-16. Can you describe your problem in more detail?
>
>   
>> Best regards,
>> Vasja
>>
>>     
>
>
>   

Re: UTF-16 problem

Posted by Doğacan Güney <do...@gmail.com>.
On 9/11/07, Vasja Ocvirk <va...@vizija.si> wrote:
> Does anyone know what to do if Nutch doesn't crawl and index web pages
> in UTF-16? Did anyone had such a problem yet?

Nutch should work with UTF-16. Can you describe your problem in more detail?

>
> Best regards,
> Vasja
>


-- 
Doğacan Güney