You are viewing a plain text version of this content. The canonical link for it is here.

Posted to solr-user@lucene.apache.org by Geepalem <na...@yahoo.com> on 2014/05/20 20:31:21 UTC

Stemming for Chinese and Japanese

Hi,

What is the filter to be used to implement stemming for Chinese and Japanese
language field types.
For English, I have used  <filter class="solr.SnowballPorterFilterFactory"
language="English" /> and its working fine.

Appreciate your help!

Thanks,
G. Naresh Kumar







--
View this message in context: http://lucene.472066.n3.nabble.com/Stemming-for-Chinese-and-Japanese-tp4137225.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Stemming for Chinese and Japanese

Posted by "T. Kuro Kurosaka" <ku...@healthline.com>.

On 05/20/2014 11:31 AM, Geepalem wrote:
> Hi,
>
> What is the filter to be used to implement stemming for Chinese and Japanese
> language field types.
> For English, I have used  <filter class="solr.SnowballPorterFilterFactory"
> language="English" /> and its working fine.
What do you mean by "working fine"?
Try analyzing this with text_en field type:
単語は何個ありますか？
This Japanese sentence for "How many tokens are there?", and the correct
answer is 5, 6 or 7, depending on how to count some compound words.
You should be seeing 10, using text_en, instead.

Try using text_ja. You will see 7.

I don't recommend to use text_cjk for Chinese, Japanese and Korean.
They are *very* different languages, and you should be using a different
analyzer for each.

StandardTokenizer just doesn't work for Chinese and Japanese at all since
there are no spaces between words in these languages.

Kuro

Re: Stemming for Chinese and Japanese

Posted by Alexandre Rafalovitch <ar...@gmail.com>.

You may want to review the series of articles about CJK support for
Solr: http://discovery-grindstone.blogspot.ca/ . Probably answers more
questions that you dare to ask.

Regards,
   Alex.
Personal website: http://www.outerthoughts.com/
Current project: http://www.solr-start.com/ - Accelerating your Solr proficiency

On Wed, May 21, 2014 at 1:31 AM, Geepalem <na...@yahoo.com> wrote:
> Hi,
>
> What is the filter to be used to implement stemming for Chinese and Japanese
> language field types.
> For English, I have used  <filter class="solr.SnowballPorterFilterFactory"
> language="English" /> and its working fine.
>
> Appreciate your help!
>
> Thanks,
> G. Naresh Kumar
>
>
>
>
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Stemming-for-Chinese-and-Japanese-tp4137225.html
> Sent from the Solr - User mailing list archive at Nabble.com.