You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Mugeesh Husain <mu...@gmail.com> on 2017/09/10 13:32:11 UTC
multi language search engine in solr
Hi
I am working on multi language search engine for english,bangla, hindi and
indonesia language. can anybody guide me how to configure solr schema.
1.) should i need to configure all the language in a single
shard/collection. ?
2.)should I need to configure separate shard/collection for each of
language ?
I am looking for the suggestion about architecture level of this project,
Please suggest and guide me to defining the schema and architecture.
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: multi language search engine in solr
Posted by Tommaso Teofili <to...@gmail.com>.
another thing to consider is what users would expect, would english user
search over english docs only ? if yes, the most important task would be to
correctly set up / create accurate per language analyzers, otherwise you
may consider to also adopt machine translation, either on the search
queries or on the resulting docs, me and a friend of mine gave a talk at
bbuzz this year [1].
My 2 cents,
Tommaso
[1] :
https://berlinbuzzwords.de/17/session/embracing-diversity-searching-over-multiple-languages
Il giorno lun 11 set 2017 alle ore 03:46 Mugeesh Husain <mu...@gmail.com>
ha scritto:
> Thank you rick for your response.
>
> The document document have sepearte of the lanaguage instead of mix of
> Arabic, English, Bengali, Hindi, Malay.
>
> I coul not find any tokenizer for Malay, can you suggest me if you know
> please.
>
>
>
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>
RE: multi language search engine in solr
Posted by Mugeesh Husain <mu...@gmail.com>.
thanks Junte Zhang, its really helpful for me
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
RE: multi language search engine in solr
Posted by Junte Zhang <Ju...@localsearch.ch>.
Having the language already separated makes it a lot easier.
You could add the language suffix (e.g. 3 letter with ISO 639-2B https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) per field where you have the different languages. Or else you could have copied an entire field to their language-analyzed fields, and hope that would be good enough for matching.
I think Malay should be very similar to Indonesian (https://wiki.apache.org/solr/LanguageAnalysis#Indonesian). However, you could extend this by adding your own dictionary (keywords) and stopwords (if that is desirable).
/JZ
-----Original Message-----
From: Mugeesh Husain [mailto:mugeesh@gmail.com]
Sent: Monday, September 11, 2017 3:46 AM
To: solr-user@lucene.apache.org
Subject: Re: multi language search engine in solr
Thank you rick for your response.
The document document have sepearte of the lanaguage instead of mix of Arabic, English, Bengali, Hindi, Malay.
I coul not find any tokenizer for Malay, can you suggest me if you know please.
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: multi language search engine in solr
Posted by Mugeesh Husain <mu...@gmail.com>.
Thank you rick for your response.
The document document have sepearte of the lanaguage instead of mix of
Arabic, English, Bengali, Hindi, Malay.
I coul not find any tokenizer for Malay, can you suggest me if you know
please.
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: multi language search engine in solr
Posted by Rick Leir <rl...@leirtech.com>.
Mugeesh,
One important question: will the typical document have a mix of English and Bangla and Hindi? If so, you would probably have them all in one collection.
Another thing to think about is the tokenizer. Are all words separated by white space? If not, then you might need to think about which tokenizer to use.
As for character sets, I think you should make sure all the inputs are in UTF-8, then there should be no problem.
There will be other things to consider but this is a start.
Cheers -- Rick
On September 10, 2017 9:32:11 AM EDT, Mugeesh Husain <mu...@gmail.com> wrote:
>Hi
>
>I am working on multi language search engine for english,bangla, hindi
>and
>indonesia language. can anybody guide me how to configure solr
>schema.
>
>1.) should i need to configure all the language in a single
>shard/collection. ?
>2.)should I need to configure separate shard/collection for each of
>language ?
>
>I am looking for the suggestion about architecture level of this
>project,
>Please suggest and guide me to defining the schema and architecture.
>
>
>
>--
>Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
--
Sorry for being brief. Alternate email is rickleir at yahoo dot com