You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by bing <ni...@hotmail.com> on 2012/01/30 04:50:20 UTC
Multilingual search in multicore solr
Hi, all,
I am going to multilingual search in multicore solr. Specifically, the
design of the solr server is like: I have several cores corresponding to
different languages, where each core has its configuration files and data.
I have following questions:
1. While indexing a document, I use ExtractingRequestHandler in Tika0.10
(embed in Solr3.5.0) and I can get a field "language_s" after indexing. Is
it possible to get the info of the "language_s" before indexing happens, so
that I can put the document in the corresponding core?
2. In searching with a query, is it possible that I can use language
detection function to determine the language code of the query, so that I
direct the query to the corresponding core?
Thanks for your suggestions.
Note: In this thread I would like to stick on multicore solr and want to
see whether the problems can be solved. Meanwhile, I am aware that
multilingual search does not necessarily need multicore solr, which I have
learned in previous thread.
http://lucene.472066.n3.nabble.com/Tika0-10-language-identifier-in-Solr3-5-0-tt3671712.html#none
--
View this message in context: http://lucene.472066.n3.nabble.com/Multilingual-search-in-multicore-solr-tp3698969p3698969.html
Sent from the Solr - User mailing list archive at Nabble.com.
Closed -- Re: Multilingual search in multicore solr
Posted by bing <ni...@hotmail.com>.
Hi, Erick,
Thanks for commenting on this thread, and I think my problem has been
solved. I might start another thread raising technical questions about using
SolrJ.
Thank you again.
Best Regards,
Bing
--
View this message in context: http://lucene.472066.n3.nabble.com/Multilingual-search-in-multicore-solr-tp3698969p3708757.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Multilingual search in multicore solr
Posted by Erick Erickson <er...@gmail.com>.
Eclipse and IntelliJ have free IDEs, both are good. Personally I prefer
IntelliJ.
Sorry, but I really can't coach you through the whole process from
the very start. I'll be happy to answer some specific questions. SolrJ
is a typical Java application, all the usual rules apply, the only tricky
part is getting your classpath pointing to the jars distributed with Solr.
Best
Erick
On Tue, Jan 31, 2012 at 10:59 PM, bing <ni...@hotmail.com> wrote:
> Hi, Erick,
>
> Thanks for your comment. Though I have some experience in Solr, I am
> completely a newbie in SolrJ, and haven't tried using SolrJ to access Solr.
> For now, I have a src package of solr3.5.0, and a SolrJ sc downloaded from
> web that I want to incorporate into Solr and have a try. How would I do to
> build and run it? Where should I put the sc in the package? Is IDE a must to
> do that?
>
> I cannot find many start-up tutorials about that, thus would be grateful if
> any suggestions and hints brought about.
>
> Best
> Bing
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Multilingual-search-in-multicore-solr-tp3698969p3705556.html
> Sent from the Solr - User mailing list archive at Nabble.com.
Re: Multilingual search in multicore solr
Posted by bing <ni...@hotmail.com>.
Hi, Erick,
Thanks for your comment. Though I have some experience in Solr, I am
completely a newbie in SolrJ, and haven't tried using SolrJ to access Solr.
For now, I have a src package of solr3.5.0, and a SolrJ sc downloaded from
web that I want to incorporate into Solr and have a try. How would I do to
build and run it? Where should I put the sc in the package? Is IDE a must to
do that?
I cannot find many start-up tutorials about that, thus would be grateful if
any suggestions and hints brought about.
Best
Bing
--
View this message in context: http://lucene.472066.n3.nabble.com/Multilingual-search-in-multicore-solr-tp3698969p3705556.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Multilingual search in multicore solr
Posted by Erick Erickson <er...@gmail.com>.
See below:
On Mon, Jan 30, 2012 at 10:16 PM, bing <ni...@hotmail.com> wrote:
> Hi, Erick Erickson,
>
> Your suggestions are sound.
>
> For (1), if I use SolrJ as the client to access Solr, then java coding
> becomes the most challenging part. Technically, I want to achieve the same
> effect with highlighting, faceting search, language detection, etc. Do you
> know some example SC that I can refer to?
>
It's actually surprisingly easy. You want to use either the
CommonsHttpSolrServer or the StreamingUpdateSolrServer
to connect to a Solr instance.
>From there, you assemble a list of SolrInputDocument and call
server.add(list).
The basic bits are about 25 lines of code.
Adding Tika in is almost equally as easy.
Don't know of any canned code lying around though.
Best
Erick
> For (2), I agree with you on the difficulty in detecting language from just
> a few words. Thus, alternatively I can suggest a set of results and let
> users to decide.
> You also mentioned score. Say, I have not so many cores, and so for every
> query I direct it to all the cores, returned with a set of scores. Is it
> confident to conclude that the highest score gives the most confidence of
> the results?
>
Absolutely not, I mislead you a bit in my original suggestion. The cores
all have independent statistics, so the scores are not comparable. Sorry
about that! This is not as bad a problem if you simply have different
*fields* per language in a single core, but still is a concern.
> Thanks.
>
> Best Regards,
> Ni Bing
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Multilingual-search-in-multicore-solr-tp3698969p3702041.html
> Sent from the Solr - User mailing list archive at Nabble.com.
Re: Multilingual search in multicore solr
Posted by bing <ni...@hotmail.com>.
Hi, Erick Erickson,
Your suggestions are sound.
For (1), if I use SolrJ as the client to access Solr, then java coding
becomes the most challenging part. Technically, I want to achieve the same
effect with highlighting, faceting search, language detection, etc. Do you
know some example SC that I can refer to?
For (2), I agree with you on the difficulty in detecting language from just
a few words. Thus, alternatively I can suggest a set of results and let
users to decide.
You also mentioned score. Say, I have not so many cores, and so for every
query I direct it to all the cores, returned with a set of scores. Is it
confident to conclude that the highest score gives the most confidence of
the results?
Thanks.
Best Regards,
Ni Bing
--
View this message in context: http://lucene.472066.n3.nabble.com/Multilingual-search-in-multicore-solr-tp3698969p3702041.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Multilingual search in multicore solr
Posted by Erick Erickson <er...@gmail.com>.
for <1>. Not that I know of. What you can do, and relatively simply
at that, is create a SolrJ program that uses Tika to parse the files
on the *client*. At that point you can do anything you'd like, including
detect language, route the document to the right core, etc. This will
also give you more control over how meta-data parsed by Tika is
mapped to your documents.
about <2>. This is hard, mostly because the queries have very
little text to analyze. Consider not worrying about it, just send the
query to all the languages in the hope that anything language
specific is scored higher.
You can do things like detect the language the browser defaults
to or ask the user to provide a "preferred language", but trying
to determine the language based on a short phrase is notoriously
hard.
Best
Erick
On Sun, Jan 29, 2012 at 10:50 PM, bing <ni...@hotmail.com> wrote:
> Hi, all,
>
> I am going to multilingual search in multicore solr. Specifically, the
> design of the solr server is like: I have several cores corresponding to
> different languages, where each core has its configuration files and data.
>
> I have following questions:
>
> 1. While indexing a document, I use ExtractingRequestHandler in Tika0.10
> (embed in Solr3.5.0) and I can get a field "language_s" after indexing. Is
> it possible to get the info of the "language_s" before indexing happens, so
> that I can put the document in the corresponding core?
>
> 2. In searching with a query, is it possible that I can use language
> detection function to determine the language code of the query, so that I
> direct the query to the corresponding core?
>
> Thanks for your suggestions.
>
> Note: In this thread I would like to stick on multicore solr and want to
> see whether the problems can be solved. Meanwhile, I am aware that
> multilingual search does not necessarily need multicore solr, which I have
> learned in previous thread.
> http://lucene.472066.n3.nabble.com/Tika0-10-language-identifier-in-Solr3-5-0-tt3671712.html#none
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Multilingual-search-in-multicore-solr-tp3698969p3698969.html
> Sent from the Solr - User mailing list archive at Nabble.com.