You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by brian beard <br...@hotmail.com> on 2007/04/23 16:18:41 UTC

solr utf 16 ?

Are there any plans to make solr UTF-16 compliant in the future?
If so, is it in the short-term or long-term?

_________________________________________________________________
MSN is giving away a trip to Vegas to see Elton John.  Enter to win today. 
http://msnconcertcontest.com?icid-nceltontagline


Re: solr utf 16 ?

Posted by brian beard <br...@hotmail.com>.
Thanks for all the comments. The conversion seems like a good alternative.

>From: "Mike Klaas" <mi...@gmail.com>
>Reply-To: solr-user@lucene.apache.org
>To: solr-user@lucene.apache.org
>Subject: Re: solr utf 16 ?
>Date: Mon, 23 Apr 2007 11:13:54 -0700
>
>On 4/23/07, brian beard <br...@hotmail.com> wrote:
>>Yes. I'm assuming if you have UTF-16 encoded data in a document that needs
>>to be added to the index, that solr would not be able to handle this?
>
>I believe that handling arbitrary encodings is on the list of future
>enhancements, but I couldn't give you a timeline.
>
>For the time being, consider that
>1. utf-8 is the "lingua franca" of xml document encoding
>2. it is very easy to convert it yourself (it would be a 3-4 line
>python commandline filter, frinstance).
>
>-Mike

_________________________________________________________________
Need a break? Find your escape route with Live Search Maps. 
http://maps.live.com/?icid=hmtag3


Re: solr utf 16 ?

Posted by Walter Underwood <wu...@netflix.com>.
UTF-16 support should not require any changes to the XML parsing.
All XML parsers are required to support that encoding. The real
change is implementing RFC 3023 (XML Media Types) so that the
encoding can be specified over HTTP.

wunder

On 4/23/07 11:13 AM, "Mike Klaas" <mi...@gmail.com> wrote:

> On 4/23/07, brian beard <br...@hotmail.com> wrote:
>> Yes. I'm assuming if you have UTF-16 encoded data in a document that needs
>> to be added to the index, that solr would not be able to handle this?
> 
> I believe that handling arbitrary encodings is on the list of future
> enhancements, but I couldn't give you a timeline.
> 
> For the time being, consider that
>  1. utf-8 is the "lingua franca" of xml document encoding
>  2. it is very easy to convert it yourself (it would be a 3-4 line
> python commandline filter, frinstance).
> 
> -Mike


Re: solr utf 16 ?

Posted by Mike Klaas <mi...@gmail.com>.
On 4/23/07, brian beard <br...@hotmail.com> wrote:
> Yes. I'm assuming if you have UTF-16 encoded data in a document that needs
> to be added to the index, that solr would not be able to handle this?

I believe that handling arbitrary encodings is on the list of future
enhancements, but I couldn't give you a timeline.

For the time being, consider that
 1. utf-8 is the "lingua franca" of xml document encoding
 2. it is very easy to convert it yourself (it would be a 3-4 line
python commandline filter, frinstance).

-Mike

Re: solr utf 16 ?

Posted by Ken Krugler <kk...@transpac.com>.
>>I'm curious what you mean by "UTF-16 complaint". Do you mean being 
>>able to handle UTF-16 encoded XML?
>>
>Yes. I'm assuming if you have UTF-16 encoded data in a document that 
>needs to be added to the index, that solr would not be able to 
>handle this?
>
I've never tried sending anything but UTF-8 to Solr, so I can't 
comment on what issues you'll run into.

But based on my experience to date, I'd strongly suggest converting 
it to UTF-8 before you post it to Solr.

-- Ken
-- 
Ken Krugler
Krugle, Inc.
+1 530-210-6378
"Find Code, Find Answers"

Re: solr utf 16 ?

Posted by brian beard <br...@hotmail.com>.
Yes. I'm assuming if you have UTF-16 encoded data in a document that needs 
to be added to the index, that solr would not be able to handle this?

>I'm curious what you mean by "UTF-16 complaint". Do you mean being able to 
>handle UTF-16 encoded XML?
>

_________________________________________________________________
Don’t quit your job – Take Classes Online and Earn your Degree in 1 year. 
Start Today! 
http://www.classesusa.com/clickcount.cfm?id=866146&goto=http%3A%2F%2Fwww.classesusa.com%2Ffeaturedschools%2Fonlinedegreesmp%2Fform-dyn1.html%3Fsplovr%3D866144


Re: solr utf 16 ?

Posted by Ken Krugler <kk...@transpac.com>.
>Are there any plans to make solr UTF-16 compliant in the future?
>If so, is it in the short-term or long-term?

I'm curious what you mean by "UTF-16 complaint". Do you mean being 
able to handle UTF-16 encoded XML?

Thanks,

-- Ken
-- 
Ken Krugler
Krugle, Inc.
+1 530-210-6378
"Find Code, Find Answers"