You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by RT <rw...@gmail.com> on 2012/01/31 09:20:01 UTC

solr utf8 for words like compagnieën?

Hi,

I am having a bit of trouble getting words with characters such as:

ë, á, ø etc into SOLR.

Programming in C++ (Using Qt's QString) I am wondering what conversion to 
apply before compiling words with such letters into the solrquery.

Is UTF8 the correct encoding?

Thanks in advance.

Kind regards,

Roland.

Re: solr utf8 for words like compagnieën?

Posted by RT <rw...@gmail.com>.
Hi Chris, Gora

thanks for the help. I am indeed writing a client conversing with solr with 
http get calls.

Using your suggestions (in particular the echoParams tip) I managed to find 
the problem.

Curiously it turns out that on sending messages I should not convert to 
Utf8 or otherwise and on receiving one has to. NOt sure if this is a Qt 
oddity but things seem to be fine now.

Thanks again,

Roland.


Chris Hostetter wrote:
> : Programming in C++ (Using Qt's QString) I am wondering what conversion to
> : apply before compiling words with such letters into the solrquery.
> 
> if you are implementing your own client and talking to Solr via HTTP then 
> how you escape/encode characters in URLs is largely dependent on how you 
> have your servlet container configured.
> 
> For example...
> 
> https://wiki.apache.org/solr/FAQ#Why_don.27t_International_Characters_Work.3F
> https://wiki.apache.org/solr/SolrTomcat#URI_Charset_Config
> 
> 
> One way to help debug what is going on is to use "echoParams=all" in your 
> request, and then look at the "requestHeader" section of the response.
> that will echo back *exactly* what solr got from the servlet container for 
> each of your input params.  If you use it with the python response writer 
> (wt=python) you will see escape sequences for any non-ascii characters -- 
> which can be helpful for verifying the problem is in how the utf8 input is 
> parsed, and not in how it is being returned to your client
> 
> there is also an "example/exampledocs/test_utf8.sh" that is helpful for 
> sanity checking these sorts of things independent of your client code.
> 
> 
> -Hoss
> 

Re: solr utf8 for words like compagnieën?

Posted by Chris Hostetter <ho...@fucit.org>.
: Programming in C++ (Using Qt's QString) I am wondering what conversion to
: apply before compiling words with such letters into the solrquery.

if you are implementing your own client and talking to Solr via HTTP then 
how you escape/encode characters in URLs is largely dependent on how you 
have your servlet container configured.

For example...

https://wiki.apache.org/solr/FAQ#Why_don.27t_International_Characters_Work.3F
https://wiki.apache.org/solr/SolrTomcat#URI_Charset_Config


One way to help debug what is going on is to use "echoParams=all" in your 
request, and then look at the "requestHeader" section of the response.
that will echo back *exactly* what solr got from the servlet container for 
each of your input params.  If you use it with the python response writer 
(wt=python) you will see escape sequences for any non-ascii characters -- 
which can be helpful for verifying the problem is in how the utf8 input is 
parsed, and not in how it is being returned to your client

there is also an "example/exampledocs/test_utf8.sh" that is helpful for 
sanity checking these sorts of things independent of your client code.


-Hoss

Re: solr utf8 for words like compagnieën?

Posted by RT <rw...@gmail.com>.
Hi,

Both Latin1 and Utf8 conversion yield the same negative results.

I get compagnieën back from SOLR as:

compagnieën

I post with: toLatin1() and retrieve from SOLR into QString with 
QString::fromLatin1()

Rather dissapointing. Any ideas as to what I may be doing wrong are very 
welcome at this stage.

Thanks,

Roland.

Gora Mohanty wrote:
> On Tue, Jan 31, 2012 at 1:50 PM, RT <rw...@gmail.com> wrote:
>> Hi,
>>
>> I am having a bit of trouble getting words with characters such as:
>>
>> ė, į, ų etc into SOLR.
>>
>> Programming in C++ (Using Qt's QString) I am wondering what conversion to
>> apply before compiling words with such letters into the solrquery.
>>
>> Is UTF8 the correct encoding?
> 
> UTF8 should be fine, though Latin1 will also work here.
> How are you getting the UTF8 for these strings? Have
> you looked at
> http://developer.qt.nokia.com/doc/qt-4.8/QString.html#converting-between-8-bit-strings-and-unicode-strings
> 
> Regards,
> Gora
> 

Re: solr utf8 for words like compagnieën?

Posted by RT <rw...@gmail.com>.
Hi Gora,

thanks a lot for the below feedback. I use toLatin1() frequently and will 
opt for that to see what it does for me.

Thanks again.

Kind regards,

Roland

Gora Mohanty wrote:
> On Tue, Jan 31, 2012 at 1:50 PM, RT <rw...@gmail.com> wrote:
>> Hi,
>>
>> I am having a bit of trouble getting words with characters such as:
>>
>> ė, į, ų etc into SOLR.
>>
>> Programming in C++ (Using Qt's QString) I am wondering what conversion to
>> apply before compiling words with such letters into the solrquery.
>>
>> Is UTF8 the correct encoding?
> 
> UTF8 should be fine, though Latin1 will also work here.
> How are you getting the UTF8 for these strings? Have
> you looked at
> http://developer.qt.nokia.com/doc/qt-4.8/QString.html#converting-between-8-bit-strings-and-unicode-strings
> 
> Regards,
> Gora
> 

Re: solr utf8 for words like compagnieën?

Posted by Gora Mohanty <go...@mimirtech.com>.
On Tue, Jan 31, 2012 at 1:50 PM, RT <rw...@gmail.com> wrote:
> Hi,
>
> I am having a bit of trouble getting words with characters such as:
>
> ė, į, ų etc into SOLR.
>
> Programming in C++ (Using Qt's QString) I am wondering what conversion to
> apply before compiling words with such letters into the solrquery.
>
> Is UTF8 the correct encoding?

UTF8 should be fine, though Latin1 will also work here.
How are you getting the UTF8 for these strings? Have
you looked at
http://developer.qt.nokia.com/doc/qt-4.8/QString.html#converting-between-8-bit-strings-and-unicode-strings

Regards,
Gora