You are viewing a plain text version of this content. The canonical link for it is here.

Posted to solr-user@lucene.apache.org by virtas <pk...@gmail.com> on 2010/10/22 17:20:01 UTC

Failing to successfully import international characters via DIH

Hi, 

wanted to share problem i have got with importing text from different
languages. All international text looks wrong on luke and on AJAX solr. 


What I see for chinese and japanese characters is this:
æ˜ ç”»ã‚„éŸ³æ¥½ãŒæ¥½ã—ã„ï¼AIã®ã‚µã‚¤ãƒ¢ãƒ³ã®ãƒ•ã‚¡ãƒ³ã§ã™ã€‚ã‚¢ãƒ€ãƒ
ã‚„ãƒžãƒƒãƒˆãŒå¥½ãã§ã™ã€‚LeeDeWyzeå„ªå‹ï¼I

Although it should be:
映画や音楽が楽しい！AIのサイモンのファンです。アダムやマットが好きです。

My setup is Ubuntu server 10.04, Tomcat6, Solr 1.4 and mysql. 

Things i have configured but with no luck:
 1. /etc/tomcat6/server.xml contains this
<Connector port="8080" protocol="HTTP/1.1" 
               connectionTimeout="20000" 
               URIEncoding="UTF-8"
               redirectPort="8443" />
 2. /etc/mysql/my.cnf contains:
 [mysqld]
  .... 
 default-character-set = utf8
  character-set-server = utf8
  
 3. /etc/solr/conf/data-config.xml 
 <dataConfig>
  <dataSource type="JdbcDataSource" 
              driver="com.mysql.jdbc.Driver"
             
url="jdbc:mysql://localhost:3306/spuvocom_spuvo?characterEncoding=UTF-8" 

               encoding = "UTF-8" />
  <document>
 4. my mysql table collation is utf8_bin	

What would you recommend changing or checking?

Thanks in advance 

-- 
View this message in context: http://lucene.472066.n3.nabble.com/Failing-to-successfully-import-international-characters-via-DIH-tp1753190p1753190.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Failing to successfully import international characters via DIH

Posted by Pradeep Singh <pk...@gmail.com>.

Holy cow, you already have this in place. I apologize. This looked exactly
the kind of problem I have solved this way.

On Fri, Oct 22, 2010 at 8:38 AM, Pradeep Singh <pk...@gmail.com> wrote:

>
>
>> What would you recommend changing or checking?
>>
>>
> Tomcat *Connector* URIEncoding. I have done this several times on tomcat,
> might be at a loss on other servers though.
>
> - Pradeep
>

Re: Failing to successfully import international characters via DIH

Posted by Pradeep Singh <pk...@gmail.com>.

>
> What would you recommend changing or checking?
>
>
Tomcat *Connector* URIEncoding. I have done this several times on tomcat,
might be at a loss on other servers though.

- Pradeep

Re: Failing to successfully import international characters via DIH

Posted by Lance Norskog <go...@gmail.com>.

CLOB is probably better for what you want.
Also, make sure the table is declared UTF-8 (or Unicode or whatever 
mysql calls it.)

virtas wrote:
> As it turns out issue was somewhere in mysql. Not sure exactly where, but
> something to do to with BLOB.
>
> Now, I changed text field from BLOB to varchar and started using
> mysql_real_escape_string in my php code and all started working just fine.
>
> Thanks for the help
>

Re: Failing to successfully import international characters via DIH

Posted by virtas <pk...@gmail.com>.

As it turns out issue was somewhere in mysql. Not sure exactly where, but
something to do to with BLOB. 

Now, I changed text field from BLOB to varchar and started using
mysql_real_escape_string in my php code and all started working just fine. 

Thanks for the help
-- 
View this message in context: http://lucene.472066.n3.nabble.com/Failing-to-successfully-import-international-characters-via-DIH-tp1753190p1770533.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Failing to successfully import international characters via DIH

Posted by Dennis Gearon <ge...@sbcglobal.net>.

Sounds like one of three things:
1/ Everything is set to UTF-*, but the content has another encoding.
2/ Something 'mirocosoftish' is adding a BOM (byte order mark) that is being incorrectly interpreted.
3/ The byte order is wrong somewhere along the way and not being translated correctly across machine/media boundaries.

You need to look at what your source is providing, directly first, before it gets into the database. Then do the following.

I would open up an editor that you KNOW outputs utf-8:

1/ Compose a web page, view it with fonts set to UTF8, that will tell you that it is really creating UTF-8 files. (Obviously use some character over 0xFF)

2/ Build an SQL query with it that inserts one record, or many, using those characters. Try commandline, server side language, and any  DBase management program also. Make the records distinct relative to where they are being inserted from.

3/ Select these records and view on a web page set to UTF-8 and see if they come out of the database OK.

4/ Import inot Solr, and view again in a browser set to UTF-8
Dennis Gearon

Signature Warning
----------------
It is always a good idea to learn from your own mistakes. It is usually a better idea to learn from others’ mistakes, so you do not have to make them yourself. from 'http://blogs.techrepublic.com.com/security/?p=4501&tag=nl.e036'

EARTH has a Right To Life,
  otherwise we all die.

--- On Fri, 10/22/10, virtas <pk...@gmail.com> wrote:

> From: virtas <pk...@gmail.com>
> Subject: Failing to successfully import international characters via DIH
> To: solr-user@lucene.apache.org
> Date: Friday, October 22, 2010, 8:20 AM
> 
> Hi, 
> 
> wanted to share problem i have got with importing text from
> different
> languages. All international text looks wrong on luke and
> on AJAX solr. 
> 
> 
> What I see for chinese and japanese characters is this:
> æ˜
> ç”»ã‚„éŸ³æ¥½ãŒæ¥½ã—ã„ï¼AIã®ã‚µã‚¤ãƒ¢ãƒ³ã®ãƒ•ã‚¡ãƒ³ã§ã™ã€‚ã‚¢ãƒ€ãƒ
> ã‚„ãƒžãƒƒãƒˆãŒå¥½ãã§ã™ã€‚LeeDeWyzeå„ªå‹ï¼I
> 
> Although it should be:
> 映画や音楽が楽しい！AIのサイモンのファンです。アダムやマットが好きです。
> 
> My setup is Ubuntu server 10.04, Tomcat6, Solr 1.4 and
> mysql. 
> 
> Things i have configured but with no luck:
>  1. /etc/tomcat6/server.xml contains this
> <Connector port="8080" protocol="HTTP/1.1" 
>            
>    connectionTimeout="20000" 
>            
>    URIEncoding="UTF-8"
>            
>    redirectPort="8443" />
>  2. /etc/mysql/my.cnf contains:
>  [mysqld]
>   .... 
>  default-character-set = utf8
>   character-set-server = utf8
>   
>  3. /etc/solr/conf/data-config.xml 
>  <dataConfig>
>   <dataSource type="JdbcDataSource" 
>              
> driver="com.mysql.jdbc.Driver"
>              
> url="jdbc:mysql://localhost:3306/spuvocom_spuvo?characterEncoding=UTF-8"
> 
> 
>            
>    encoding = "UTF-8" />
>   <document>
>  4. my mysql table collation is utf8_bin   
> 
> 
> What would you recommend changing or checking?
> 
> Thanks in advance 
> 
> -- 
> View this message in context: http://lucene.472066.n3.nabble.com/Failing-to-successfully-import-international-characters-via-DIH-tp1753190p1753190.html
> Sent from the Solr - User mailing list archive at
> Nabble.com.
>