You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Kuntal Ganguly <ga...@gmail.com> on 2015/05/07 20:23:49 UTC

Solr Multilingual Indexing with one field- Guidance

Our current production index size is 1.5 TB with 3 shards. Currently we
have the following field type:

<fieldType name="text_ngram" class="solr.TextField"
positionIncrementGap="100">

<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.CustomNGramFilterFactory" minGramSize="3"
maxGramSize="30" preserveOriginal="true"/>
</analyzer>
</fieldType>

And the above field type is working well for the US and English language
clients.

Now we have some new Chinese and Japanese client ,so after google
http://www.basistech.com/indexing-strategies-for-multilingual-search-with-solr-and-rosette/

https://docs.lucidworks.com/display/lweug/Multilingual+Indexing+and+Search

for best approach for multilingual index,there seems to be pros/cons
associated with every approach.

Then i tried RnD with a single field approach and here's my new field type:

<fieldType name="text_multi" class="solr.TextField"
positionIncrementGap="100">

<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.CJKWidthFilterFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.CJKBigramFilterFactory"/>
</analyzer>
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.CJKWidthFilterFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.CJKBigramFilterFactory"/>
<filter class="solr.CustomNGramFilterFactory" minGramSize="3"
maxGramSize="30" preserveOriginal="true"/>
</analyzer>
</fieldType>

I have kept the same tokenizer, only changed the filters.And it is working
well with all existing search /use-case for English documents as well as
new use case for Chinese/Japanese documents.

Now i have the following questions to the Solr experts/developer:

1) Is this a correct approach to do it? Or i'm missing something?

2) Can you give me an example where there will be problem with this above
new field type? A use-case/scenario with example will be very helpful.

3) Also is there any problem in future with different clients coming up?

Please provide some guidance

Re: Solr Multilingual Indexing with one field- Guidance

Posted by Alessandro Benedetti <be...@gmail.com>.
Is it possible to know a little bit more about the nature of that
multi-lingual field ?
I can see the keywordTokenizer and then a lot of grams calculated from that
token .
What is that field used for ?

2015-05-07 19:23 GMT+01:00 Kuntal Ganguly <ga...@gmail.com>:

> Our current production index size is 1.5 TB with 3 shards. Currently we
> have the following field type:
>
> <fieldType name="text_ngram" class="solr.TextField"
> positionIncrementGap="100">
>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="index">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.CustomNGramFilterFactory" minGramSize="3"
> maxGramSize="30" preserveOriginal="true"/>
> </analyzer>
> </fieldType>
>
> And the above field type is working well for the US and English language
> clients.
>
> Now we have some new Chinese and Japanese client ,so after google
>
> http://www.basistech.com/indexing-strategies-for-multilingual-search-with-solr-and-rosette/
>
> https://docs.lucidworks.com/display/lweug/Multilingual+Indexing+and+Search
>
> for best approach for multilingual index,there seems to be pros/cons
> associated with every approach.
>
> Then i tried RnD with a single field approach and here's my new field type:
>
> <fieldType name="text_multi" class="solr.TextField"
> positionIncrementGap="100">
>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.CJKWidthFilterFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.CJKBigramFilterFactory"/>
> </analyzer>
> <analyzer type="index">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.CJKWidthFilterFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.CJKBigramFilterFactory"/>
> <filter class="solr.CustomNGramFilterFactory" minGramSize="3"
> maxGramSize="30" preserveOriginal="true"/>
> </analyzer>
> </fieldType>
>
> I have kept the same tokenizer, only changed the filters.And it is working
> well with all existing search /use-case for English documents as well as
> new use case for Chinese/Japanese documents.
>
> Now i have the following questions to the Solr experts/developer:
>
> 1) Is this a correct approach to do it? Or i'm missing something?
>
> 2) Can you give me an example where there will be problem with this above
> new field type? A use-case/scenario with example will be very helpful.
>
> 3) Also is there any problem in future with different clients coming up?
>
> Please provide some guidance
>



-- 
--------------------------

Benedetti Alessandro
Visiting card : http://about.me/alessandro_benedetti

"Tyger, tyger burning bright
In the forests of the night,
What immortal hand or eye
Could frame thy fearful symmetry?"

William Blake - Songs of Experience -1794 England

Re: Solr Multilingual Indexing with one field- Guidance

Posted by Alexandre Rafalovitch <ar...@gmail.com>.
On 8 May 2015 at 04:23, Kuntal Ganguly <ga...@gmail.com> wrote:
> Please provide some guidance

This question comes up a lot on the list and has been discussed
multiple times. Did you try searching the mailing list for past
discussions? E.g. something like:
http://search-lucene.com/?q=multilingual+indexing+single+field&fc_project=Solr&fc_type=mail+_hash_+user

Regards,
  Alex.


----
Solr Analyzers, Tokenizers, Filters, URPs and even a newsletter:
http://www.solr-start.com/

Re: Solr Multilingual Indexing with one field- Guidance

Posted by TK Solr <tk...@sonic.net>.
On 5/7/15, 11:23 AM, Kuntal Ganguly wrote:
> 1) Is this a correct approach to do it? Or i'm missing something?
Does the user wants to see the documents that he/she doesn't understand?
The words such as "doctor", "taxi", etc. are common among many languages in Europe.
Would the Spanish user wants to see English documents?
Of course this issue can be worked-around by having a separate language field.

How do you handle word collision among languages ?
"kind" in German means "child" in English. If a German user search for articles
about children, they will find lots of unrelated English
articles about someone being kind.
This one too can be worked-around by having a language field.

By default, Solr/Lucene hits are sort by the relevancy scores and
the score calculation uses IDF. If a search term appears in many documents,
the score is low. Because virtually all German documents have "die", the particle,
the score of the English word "die" will be low also.
>
> 2) Can you give me an example where there will be problem with this above
> new field type? A use-case/scenario with example will be very helpful.

If you have lots of Japanese documents indexed, try searching "京都" (Kyoto).
You will find many documents about Tokyo (東京) because the government
of the metropolitan Tokyo area is spelled as "東京都" = Tokyo Capital, which
generates two bigrams, 東京 and 京都.

Kuro