You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by nibing <ni...@hotmail.com> on 2012/01/19 09:22:39 UTC
Tika0.10 language identifier in Solr3.5.0
Hi, all,
I am using Solr3.5.0 which applies Tika0.10 to do language detection,
and I have a couple of questions about this function.
1. I can see the outcome of the language detection in a field
"language_s". But what action will be taken according to the different
language code? How to configure?
2. Currently the language detection only happens in indexing. Is it
possible to use the function in searching as well? How to configure?
Many thanks.
Best Regards
Ni, Bing
Re: Tika0.10 language identifier in Solr3.5.0
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Hi,
I think guessing the language based purely on query string is OK *if* you are OK it not being very accurate and finding ways to work around that, say by giving users the options to switch to another language easily, allowing them to easily select a default language for them in the future, etc.
Otis
----
Performance Monitoring SaaS for Solr - http://sematext.com/spm/solr-performance-monitoring/index.html
----- Original Message -----
> From: nibing <ni...@hotmail.com>
> To: solr-user@lucene.apache.org
> Cc:
> Sent: Thursday, January 19, 2012 10:35 PM
> Subject: RE: Tika0.10 language identifier in Solr3.5.0
>
>
> Hi, Jan Høydahl You are right. I am hoping to detect the language of a query,
> so that the serarching can be done according to the language detected. Since
> people often type a few words, which is too few to detect, then it is hard to do
> that. Let me describe a little bit about the solr server in my design. It
> consists of several cores, corresponding to the several languages, which is
> built during indexing. Since language detection in indexing can be done with
> Tika identifier, then we are currently OK. But the problem is about searching. I
> want to do language detection first before do searching in the individual cores.
> In the case that detection result is ambiguous and several languages are
> returned, we probably returns a set of results, and let user to decide which
> language set of results they want to look into. In general, it is just the same
> with the language supported by google. Do you have some suggestions if I want to
> achieve multilingual search described as above? Thank you.
> Best Regards
> Ni, Bing
>
>> Subject: Re: Tika0.10 language identifier in Solr3.5.0
>> From: jan.asf@cominvent.com
>> Date: Thu, 19 Jan 2012 12:31:01 +0100
>> To: solr-user@lucene.apache.org
>>
>> Hi,
>>
>> You may use the string as you choose, for instance filtering
> (fq=language_s:en) or for faceting (facet.field=language_s). What are you
> looking to do?
>>
>> What would you like to detect on the query side? The language of the search
> string? That is very hard since people type very few words into the search box.
>>
>> --
>> Jan Høydahl, search solution architect
>> Cominvent AS - www.cominvent.com
>> Solr Training - www.solrtraining.com
>>
>> On 19. jan. 2012, at 09:22, nibing wrote:
>>
>> >
>> > Hi, all,
>> >
>> >
>> >
>> > I am using Solr3.5.0 which applies Tika0.10 to do language detection,
>> > and I have a couple of questions about this function.
>> >
>> >
>> >
>> > 1. I can see the outcome of the language detection in a field
>> > "language_s". But what action will be taken according to the
> different
>> > language code? How to configure?
>> >
>> >
>> >
>> > 2. Currently the language detection only happens in indexing. Is it
>> > possible to use the function in searching as well? How to configure?
>> >
>> >
>> >
>> > Many thanks.
>> >
>> >
>> > Best Regards
>> >
>> > Ni, Bing
>> >
>>
>
RE: Tika0.10 language identifier in Solr3.5.0
Posted by nibing <ni...@hotmail.com>.
Hi, Ted Dunning,
Thank you for your reply. I can understand your point on putting a "language_s" field and then keeping all the files together, which speed-up searching.
But then there occurs a problem of using analyzer in indexing. I assume files encoded in different language should be handled using different analyzers, i.e. different tokenizers and filters. Can you elaborate a little bit on the design that you propose, especially in how files encoded in different languages can be handled during the indexing?
Thank you.
Best Regards
Ni, Bing
Best Regards
Ni, Bing
> From: ted.dunning@gmail.com
> Date: Fri, 20 Jan 2012 03:55:48 +0000
> Subject: Re: Tika0.10 language identifier in Solr3.5.0
> To: solr-user@lucene.apache.org
>
> Normally this is done by putting a field on each document rather than
> separating the documents into separate corpora. Keeping them together
> makes the final search faster.
>
> At query time, you can add all of the language keys that you think are
> relevant based on your language id applied to the query and then group the
> results on those keys so that users can inspect different language results.
> If you require the correct language key, you should get pretty good
> retrieval speed.
>
> On Fri, Jan 20, 2012 at 3:35 AM, nibing <ni...@hotmail.com> wrote:
>
> >
> > Hi, Jan Høydahl You are right. I am hoping to detect the language of a
> > query, so that the serarching can be done according to the language
> > detected. Since people often type a few words, which is too few to detect,
> > then it is hard to do that. Let me describe a little bit about the solr
> > server in my design. It consists of several cores, corresponding to the
> > several languages, which is built during indexing. Since language detection
> > in indexing can be done with Tika identifier, then we are currently OK. But
> > the problem is about searching. I want to do language detection first
> > before do searching in the individual cores. In the case that detection
> > result is ambiguous and several languages are returned, we probably returns
> > a set of results, and let user to decide which language set of results they
> > want to look into. In general, it is just the same with the language
> > supported by google. Do you have some suggestions if I want to achieve
> > multilingual search described as above? Thank you.
> > Best Regards
> > Ni, Bing
> >
> > > Subject: Re: Tika0.10 language identifier in Solr3.5.0
> > > From: jan.asf@cominvent.com
> > > Date: Thu, 19 Jan 2012 12:31:01 +0100
> > > To: solr-user@lucene.apache.org
> > >
> > > Hi,
> > >
> > > You may use the string as you choose, for instance filtering
> > (fq=language_s:en) or for faceting (facet.field=language_s). What are you
> > looking to do?
> > >
> > > What would you like to detect on the query side? The language of the
> > search string? That is very hard since people type very few words into the
> > search box.
> > >
> > > --
> > > Jan Høydahl, search solution architect
> > > Cominvent AS - www.cominvent.com
> > > Solr Training - www.solrtraining.com
> > >
> > > On 19. jan. 2012, at 09:22, nibing wrote:
> > >
> > > >
> > > > Hi, all,
> > > >
> > > >
> > > >
> > > > I am using Solr3.5.0 which applies Tika0.10 to do language detection,
> > > > and I have a couple of questions about this function.
> > > >
> > > >
> > > >
> > > > 1. I can see the outcome of the language detection in a field
> > > > "language_s". But what action will be taken according to the different
> > > > language code? How to configure?
> > > >
> > > >
> > > >
> > > > 2. Currently the language detection only happens in indexing. Is it
> > > > possible to use the function in searching as well? How to configure?
> > > >
> > > >
> > > >
> > > > Many thanks.
> > > >
> > > >
> > > > Best Regards
> > > >
> > > > Ni, Bing
> > > >
> > >
> >
Re: Tika0.10 language identifier in Solr3.5.0
Posted by Ted Dunning <te...@gmail.com>.
Write a tokenizer that does language ID and then picks which tokenizer to
use. Then record the language in the language id field.
What is there to elaborate?
On Fri, Jan 20, 2012 at 1:58 AM, nibing <ni...@hotmail.com> wrote:
> But then there occurs a problem of using analyzer in indexing. I assume
> files encoded in different language should be handled using different
> analyzers, i.e. different tokenizers and filters. Can you elaborate a
> little bit on the design that you propose, especially in how files encoded
> in different languages can be handled during the indexing?
RE: Tika0.10 language identifier in Solr3.5.0
Posted by nibing <ni...@hotmail.com>.
Hi, Ted Dunning,
Thank you for your reply. I can understand your point on putting a "language_s" field and then keeping all the files together, which speed-up searching.
But then there occurs a problem of using analyzer in indexing. I assume files encoded in different language should be handled using different analyzers, i.e. different tokenizers and filters. Can you elaborate a little bit on the design that you propose, especially in how files encoded in different languages can be handled during the indexing?
Thank you.
Best Regards
Ni, Bing
> From: ted.dunning@gmail.com
> Date: Fri, 20 Jan 2012 03:55:48 +0000
> Subject: Re: Tika0.10 language identifier in Solr3.5.0
> To: solr-user@lucene.apache.org
>
> Normally this is done by putting a field on each document rather than
> separating the documents into separate corpora. Keeping them together
> makes the final search faster.
>
> At query time, you can add all of the language keys that you think are
> relevant based on your language id applied to the query and then group the
> results on those keys so that users can inspect different language results.
> If you require the correct language key, you should get pretty good
> retrieval speed.
>
> On Fri, Jan 20, 2012 at 3:35 AM, nibing <ni...@hotmail.com> wrote:
>
> >
> > Hi, Jan Høydahl You are right. I am hoping to detect the language of a
> > query, so that the serarching can be done according to the language
> > detected. Since people often type a few words, which is too few to detect,
> > then it is hard to do that. Let me describe a little bit about the solr
> > server in my design. It consists of several cores, corresponding to the
> > several languages, which is built during indexing. Since language detection
> > in indexing can be done with Tika identifier, then we are currently OK. But
> > the problem is about searching. I want to do language detection first
> > before do searching in the individual cores. In the case that detection
> > result is ambiguous and several languages are returned, we probably returns
> > a set of results, and let user to decide which language set of results they
> > want to look into. In general, it is just the same with the language
> > supported by google. Do you have some suggestions if I want to achieve
> > multilingual search described as above? Thank you.
> > Best Regards
> > Ni, Bing
> >
> > > Subject: Re: Tika0.10 language identifier in Solr3.5.0
> > > From: jan.asf@cominvent.com
> > > Date: Thu, 19 Jan 2012 12:31:01 +0100
> > > To: solr-user@lucene.apache.org
> > >
> > > Hi,
> > >
> > > You may use the string as you choose, for instance filtering
> > (fq=language_s:en) or for faceting (facet.field=language_s). What are you
> > looking to do?
> > >
> > > What would you like to detect on the query side? The language of the
> > search string? That is very hard since people type very few words into the
> > search box.
> > >
> > > --
> > > Jan Høydahl, search solution architect
> > > Cominvent AS - www.cominvent.com
> > > Solr Training - www.solrtraining.com
> > >
> > > On 19. jan. 2012, at 09:22, nibing wrote:
> > >
> > > >
> > > > Hi, all,
> > > >
> > > >
> > > >
> > > > I am using Solr3.5.0 which applies Tika0.10 to do language detection,
> > > > and I have a couple of questions about this function.
> > > >
> > > >
> > > >
> > > > 1. I can see the outcome of the language detection in a field
> > > > "language_s". But what action will be taken according to the different
> > > > language code? How to configure?
> > > >
> > > >
> > > >
> > > > 2. Currently the language detection only happens in indexing. Is it
> > > > possible to use the function in searching as well? How to configure?
> > > >
> > > >
> > > >
> > > > Many thanks.
> > > >
> > > >
> > > > Best Regards
> > > >
> > > > Ni, Bing
> > > >
> > >
> >
Re: Tika0.10 language identifier in Solr3.5.0
Posted by Ted Dunning <te...@gmail.com>.
Jan's point that keeping different fields can make some statistical issues
more correct is sound.
The basic idea is that a common word in a rare language should be treated
as a common word if you are working in that language. The simplest way to
make that happen is by having a different field for each language. Since
Lucene itself doesn't much care which fields you have, this is a nice
option. It leads to complicated looking queries if you don't know what
language the query is in since you have to have as many versions of the
query as you have languages in your corpus, but the performance should be
about the same as if all languages go into a single field. If you have a
moderately sharp idea about what language you might have, then the query
doesn't have to be all that huge.
The index size should also be about the same for either approach.
As such, I think I would go with what Jan suggested.
On Sun, Jan 22, 2012 at 5:45 PM, Erick Erickson <er...@gmail.com>wrote:
> Would "doing the right thing" include firing the results at different
> fields based on the language detected? Your answer to Jan
> seems to indicate not, in which case my original comments
> stand. The main point is that mixing all the *results* of the
> analysis chains for multiple languages into a single field
> will likely result in "interesting" behavior. Not to say it won't
> be satisfactory in your situation, but there are edge cases.
>
> Best
> Erick
>
> On Fri, Jan 20, 2012 at 9:15 AM, Ted Dunning <te...@gmail.com>
> wrote:
> > I think you misunderstood what I am suggesting.
> >
> > I am suggesting an analyzer that detects the language and then "does the
> > right thing" according to the language it finds. As such, it would
> > tokenize and stem English according to English rules, German by German
> > rules and would probably do a sliding bigram window in Japanese and
> Chinese.
> >
> > On Fri, Jan 20, 2012 at 8:54 AM, Erick Erickson <erickerickson@gmail.com
> >wrote:
> >
> >> bq: Why not have a polyglot analyzer
> >>
> >> That could work, but it makes some compromises and assumes that your
> >> languages are "close enough", I have absolutely no clue how that would
> >> work for English and Chinese say.
> >>
> >> But it also introduces inconsistencies. Take stemming. Even though you
> >> could easily stem in the correct language, throwing all those stems
> >> into the same filed can produce interesting results at search time since
> >> you run the risk of hitting something produced by one of the other
> >> analysis chains.
> >>
>
Re: Tika0.10 language identifier in Solr3.5.0
Posted by Erick Erickson <er...@gmail.com>.
Would "doing the right thing" include firing the results at different
fields based on the language detected? Your answer to Jan
seems to indicate not, in which case my original comments
stand. The main point is that mixing all the *results* of the
analysis chains for multiple languages into a single field
will likely result in "interesting" behavior. Not to say it won't
be satisfactory in your situation, but there are edge cases.
Best
Erick
On Fri, Jan 20, 2012 at 9:15 AM, Ted Dunning <te...@gmail.com> wrote:
> I think you misunderstood what I am suggesting.
>
> I am suggesting an analyzer that detects the language and then "does the
> right thing" according to the language it finds. As such, it would
> tokenize and stem English according to English rules, German by German
> rules and would probably do a sliding bigram window in Japanese and Chinese.
>
> On Fri, Jan 20, 2012 at 8:54 AM, Erick Erickson <er...@gmail.com>wrote:
>
>> bq: Why not have a polyglot analyzer
>>
>> That could work, but it makes some compromises and assumes that your
>> languages are "close enough", I have absolutely no clue how that would
>> work for English and Chinese say.
>>
>> But it also introduces inconsistencies. Take stemming. Even though you
>> could easily stem in the correct language, throwing all those stems
>> into the same filed can produce interesting results at search time since
>> you run the risk of hitting something produced by one of the other
>> analysis chains.
>>
Re: Tika0.10 language identifier in Solr3.5.0
Posted by Ted Dunning <te...@gmail.com>.
The TF-IDF argument is a reasonable one.
On Fri, Jan 20, 2012 at 5:33 PM, Jan Høydahl <ja...@cominvent.com> wrote:
> Another benefit with separate field per lang is that TF/IDF stats gets
> correct for each individual language.
> Also if you KNOW the query language, you can target THAT field alone, but
> if you don't know, you can throw the query at multiple fields, which will
> each get proper analysis (at the risk of lower precision)
>
> The only case where I would prefer having one single field for all
> languages is if my search app needs to support a large amount of languages,
> such as a wide web crawl with 100 languages crawled. The way FAST supported
> this was to go lemmatization by index expansion instead of reduction or
> stemming - then you can easily support full linguistics for 100 languages,
> indexed in the same field.
>
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.cominvent.com
> Solr Training - www.solrtraining.com
>
> On 20. jan. 2012, at 18:15, Ted Dunning wrote:
>
> > I think you misunderstood what I am suggesting.
> >
> > I am suggesting an analyzer that detects the language and then "does the
> > right thing" according to the language it finds. As such, it would
> > tokenize and stem English according to English rules, German by German
> > rules and would probably do a sliding bigram window in Japanese and
> Chinese.
> >
> > On Fri, Jan 20, 2012 at 8:54 AM, Erick Erickson <erickerickson@gmail.com
> >wrote:
> >
> >> bq: Why not have a polyglot analyzer
> >>
> >> That could work, but it makes some compromises and assumes that your
> >> languages are "close enough", I have absolutely no clue how that would
> >> work for English and Chinese say.
> >>
> >> But it also introduces inconsistencies. Take stemming. Even though you
> >> could easily stem in the correct language, throwing all those stems
> >> into the same filed can produce interesting results at search time since
> >> you run the risk of hitting something produced by one of the other
> >> analysis chains.
> >>
>
>
Re: Tika0.10 language identifier in Solr3.5.0
Posted by Jan Høydahl <ja...@cominvent.com>.
Another benefit with separate field per lang is that TF/IDF stats gets correct for each individual language.
Also if you KNOW the query language, you can target THAT field alone, but if you don't know, you can throw the query at multiple fields, which will each get proper analysis (at the risk of lower precision)
The only case where I would prefer having one single field for all languages is if my search app needs to support a large amount of languages, such as a wide web crawl with 100 languages crawled. The way FAST supported this was to go lemmatization by index expansion instead of reduction or stemming - then you can easily support full linguistics for 100 languages, indexed in the same field.
--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com
Solr Training - www.solrtraining.com
On 20. jan. 2012, at 18:15, Ted Dunning wrote:
> I think you misunderstood what I am suggesting.
>
> I am suggesting an analyzer that detects the language and then "does the
> right thing" according to the language it finds. As such, it would
> tokenize and stem English according to English rules, German by German
> rules and would probably do a sliding bigram window in Japanese and Chinese.
>
> On Fri, Jan 20, 2012 at 8:54 AM, Erick Erickson <er...@gmail.com>wrote:
>
>> bq: Why not have a polyglot analyzer
>>
>> That could work, but it makes some compromises and assumes that your
>> languages are "close enough", I have absolutely no clue how that would
>> work for English and Chinese say.
>>
>> But it also introduces inconsistencies. Take stemming. Even though you
>> could easily stem in the correct language, throwing all those stems
>> into the same filed can produce interesting results at search time since
>> you run the risk of hitting something produced by one of the other
>> analysis chains.
>>
RE: Tika0.10 language identifier in Solr3.5.0
Posted by nibing <ni...@hotmail.com>.
Hi, This is exactly what I hope you can elaborate on - analyzer that detects the language and then analyze accordingly. How to do that? Thank you.
Best Regards
Ni, Bing
> From: ted.dunning@gmail.com
> Date: Fri, 20 Jan 2012 09:15:30 -0800
> Subject: Re: Tika0.10 language identifier in Solr3.5.0
> To: solr-user@lucene.apache.org
>
> I think you misunderstood what I am suggesting.
>
> I am suggesting an analyzer that detects the language and then "does the
> right thing" according to the language it finds. As such, it would
> tokenize and stem English according to English rules, German by German
> rules and would probably do a sliding bigram window in Japanese and Chinese.
>
> On Fri, Jan 20, 2012 at 8:54 AM, Erick Erickson <er...@gmail.com>wrote:
>
> > bq: Why not have a polyglot analyzer
> >
> > That could work, but it makes some compromises and assumes that your
> > languages are "close enough", I have absolutely no clue how that would
> > work for English and Chinese say.
> >
> > But it also introduces inconsistencies. Take stemming. Even though you
> > could easily stem in the correct language, throwing all those stems
> > into the same filed can produce interesting results at search time since
> > you run the risk of hitting something produced by one of the other
> > analysis chains.
> >
Re: Tika0.10 language identifier in Solr3.5.0
Posted by Ted Dunning <te...@gmail.com>.
I think you misunderstood what I am suggesting.
I am suggesting an analyzer that detects the language and then "does the
right thing" according to the language it finds. As such, it would
tokenize and stem English according to English rules, German by German
rules and would probably do a sliding bigram window in Japanese and Chinese.
On Fri, Jan 20, 2012 at 8:54 AM, Erick Erickson <er...@gmail.com>wrote:
> bq: Why not have a polyglot analyzer
>
> That could work, but it makes some compromises and assumes that your
> languages are "close enough", I have absolutely no clue how that would
> work for English and Chinese say.
>
> But it also introduces inconsistencies. Take stemming. Even though you
> could easily stem in the correct language, throwing all those stems
> into the same filed can produce interesting results at search time since
> you run the risk of hitting something produced by one of the other
> analysis chains.
>
Re: Tika0.10 language identifier in Solr3.5.0
Posted by Erick Erickson <er...@gmail.com>.
bq: Why not have a polyglot analyzer
That could work, but it makes some compromises and assumes that your
languages are "close enough", I have absolutely no clue how that would
work for English and Chinese say.
But it also introduces inconsistencies. Take stemming. Even though you
could easily stem in the correct language, throwing all those stems
into the same filed can produce interesting results at search time since
you run the risk of hitting something produced by one of the other
analysis chains.
Which may be OK from your perspective, but some will find the decreased
precision unacceptable.
The other issue is that it's really tough to understand what the language of
a query is because it's so very short. You can sometimes have your app
fake it with any language IDs set in the browser, but they aren't reliable. If
you index into language-specific fields, you can fire the same query
at all the different language-specific fields with edismax-style
handlers.
Your spelling suggestions will be "interesting" in that scenario.
The point is that putting all the text in a single field is reasonable under
some circumstances but totally unacceptable in others. It depends (tm).
If you're using Western languages, you might even be able to get away
with having a single analyzer that normalized all the accents
(one of the FoldingFilters) and not have to bother with anything else.
On Fri, Jan 20, 2012 at 8:19 AM, Ted Dunning <te...@gmail.com> wrote:
> Otis,
>
> Can you say why there needs to be a field per language? Why not have a
> polyglot analyzer?
>
> On Fri, Jan 20, 2012 at 7:29 AM, Otis Gospodnetic <
> otis_gospodnetic@yahoo.com> wrote:
>
>> Ni, Bing
>>
>> I believe you will need to pre-define fields for all languages you want to
>> handle and specify an appropriate language-specific analyzer for each of
>> those fields.
>> This also means that if you encounter a new language, you will need to
>> adjust your schema to support a new language. Of course, your Lang
>> Identifier will also need to be trained to recognize that language.
>>
>> I know ElasticSearch allows you to specify a field analyzer at index time,
>> which is very handy. I don't think you can do that with Solr today, but
>> maybe I'm forgetting something! :)
>>
>> Otis
>> ----
>> Performance Monitoring SaaS for Solr -
>> http://sematext.com/spm/solr-performance-monitoring/index.html
>>
>>
>> ----- Original Message -----
>> > From: nibing <ni...@hotmail.com>
>> > To: solr-user@lucene.apache.org
>> > Cc:
>> > Sent: Friday, January 20, 2012 1:51 AM
>> > Subject: RE: Tika0.10 language identifier in Solr3.5.0
>> >
>> >
>> > Hi, Ted Dunning,
>> >
>> > Thank you for your reply. I can understand your point on putting a
>> > "language_s" field and then keeping all the files together, which
>> > speed-up searching.
>> >
>> > But then there occurs a problem of using analyzer in indexing. I assume
>> files
>> > encoded in different language should be handled using different
>> analyzers, i.e.
>> > different tokenizers and filters. Can you elaborate a little bit on the
>> design
>> > that you propose, especially in how files encoded in different languages
>> can be
>> > handled during the indexing?
>> > Thank you.
>> >
>> > Best Regards
>> >
>> > Ni, Bing
>> >
>> >
>> >> From: ted.dunning@gmail.com
>> >> Date: Fri, 20 Jan 2012 03:55:48 +0000
>> >> Subject: Re: Tika0.10 language identifier in Solr3.5.0
>> >> To: solr-user@lucene.apache.org
>> >>
>> >> Normally this is done by putting a field on each document rather than
>> >> separating the documents into separate corpora. Keeping them together
>> >> makes the final search faster.
>> >>
>> >> At query time, you can add all of the language keys that you think are
>> >> relevant based on your language id applied to the query and then group
>> the
>> >> results on those keys so that users can inspect different language
>> results.
>> >> If you require the correct language key, you should get pretty good
>> >> retrieval speed.
>> >>
>> >> On Fri, Jan 20, 2012 at 3:35 AM, nibing <ni...@hotmail.com>
>> > wrote:
>> >>
>> >> >
>> >> > Hi, Jan Høydahl You are right. I am hoping to detect the language
>> of
>> > a
>> >> > query, so that the serarching can be done according to the language
>> >> > detected. Since people often type a few words, which is too few to
>> > detect,
>> >> > then it is hard to do that. Let me describe a little bit about the
>> > solr
>> >> > server in my design. It consists of several cores, corresponding to
>> > the
>> >> > several languages, which is built during indexing. Since language
>> > detection
>> >> > in indexing can be done with Tika identifier, then we are currently
>> > OK. But
>> >> > the problem is about searching. I want to do language detection first
>> >> > before do searching in the individual cores. In the case that
>> > detection
>> >> > result is ambiguous and several languages are returned, we probably
>> > returns
>> >> > a set of results, and let user to decide which language set of
>> results
>> > they
>> >> > want to look into. In general, it is just the same with the language
>> >> > supported by google. Do you have some suggestions if I want to
>> achieve
>> >> > multilingual search described as above? Thank you.
>> >> > Best Regards
>> >> > Ni, Bing
>> >> >
>> >> > > Subject: Re: Tika0.10 language identifier in Solr3.5.0
>> >> > > From: jan.asf@cominvent.com
>> >> > > Date: Thu, 19 Jan 2012 12:31:01 +0100
>> >> > > To: solr-user@lucene.apache.org
>> >> > >
>> >> > > Hi,
>> >> > >
>> >> > > You may use the string as you choose, for instance filtering
>> >> > (fq=language_s:en) or for faceting (facet.field=language_s). What are
>> > you
>> >> > looking to do?
>> >> > >
>> >> > > What would you like to detect on the query side? The language of
>> > the
>> >> > search string? That is very hard since people type very few words
>> into
>> > the
>> >> > search box.
>> >> > >
>> >> > > --
>> >> > > Jan Høydahl, search solution architect
>> >> > > Cominvent AS - www.cominvent.com
>> >> > > Solr Training - www.solrtraining.com
>> >> > >
>> >> > > On 19. jan. 2012, at 09:22, nibing wrote:
>> >> > >
>> >> > > >
>> >> > > > Hi, all,
>> >> > > >
>> >> > > >
>> >> > > >
>> >> > > > I am using Solr3.5.0 which applies Tika0.10 to do language
>> > detection,
>> >> > > > and I have a couple of questions about this function.
>> >> > > >
>> >> > > >
>> >> > > >
>> >> > > > 1. I can see the outcome of the language detection in a
>> > field
>> >> > > > "language_s". But what action will be taken
>> > according to the different
>> >> > > > language code? How to configure?
>> >> > > >
>> >> > > >
>> >> > > >
>> >> > > > 2. Currently the language detection only happens in
>> > indexing. Is it
>> >> > > > possible to use the function in searching as well? How to
>> > configure?
>> >> > > >
>> >> > > >
>> >> > > >
>> >> > > > Many thanks.
>> >> > > >
>> >> > > >
>> >> > > > Best Regards
>> >> > > >
>> >> > > > Ni, Bing
>> >> > > >
>> >> > >
>> >> >
>> >
>>
Re: Tika0.10 language identifier in Solr3.5.0
Posted by Ted Dunning <te...@gmail.com>.
Otis,
Can you say why there needs to be a field per language? Why not have a
polyglot analyzer?
On Fri, Jan 20, 2012 at 7:29 AM, Otis Gospodnetic <
otis_gospodnetic@yahoo.com> wrote:
> Ni, Bing
>
> I believe you will need to pre-define fields for all languages you want to
> handle and specify an appropriate language-specific analyzer for each of
> those fields.
> This also means that if you encounter a new language, you will need to
> adjust your schema to support a new language. Of course, your Lang
> Identifier will also need to be trained to recognize that language.
>
> I know ElasticSearch allows you to specify a field analyzer at index time,
> which is very handy. I don't think you can do that with Solr today, but
> maybe I'm forgetting something! :)
>
> Otis
> ----
> Performance Monitoring SaaS for Solr -
> http://sematext.com/spm/solr-performance-monitoring/index.html
>
>
> ----- Original Message -----
> > From: nibing <ni...@hotmail.com>
> > To: solr-user@lucene.apache.org
> > Cc:
> > Sent: Friday, January 20, 2012 1:51 AM
> > Subject: RE: Tika0.10 language identifier in Solr3.5.0
> >
> >
> > Hi, Ted Dunning,
> >
> > Thank you for your reply. I can understand your point on putting a
> > "language_s" field and then keeping all the files together, which
> > speed-up searching.
> >
> > But then there occurs a problem of using analyzer in indexing. I assume
> files
> > encoded in different language should be handled using different
> analyzers, i.e.
> > different tokenizers and filters. Can you elaborate a little bit on the
> design
> > that you propose, especially in how files encoded in different languages
> can be
> > handled during the indexing?
> > Thank you.
> >
> > Best Regards
> >
> > Ni, Bing
> >
> >
> >> From: ted.dunning@gmail.com
> >> Date: Fri, 20 Jan 2012 03:55:48 +0000
> >> Subject: Re: Tika0.10 language identifier in Solr3.5.0
> >> To: solr-user@lucene.apache.org
> >>
> >> Normally this is done by putting a field on each document rather than
> >> separating the documents into separate corpora. Keeping them together
> >> makes the final search faster.
> >>
> >> At query time, you can add all of the language keys that you think are
> >> relevant based on your language id applied to the query and then group
> the
> >> results on those keys so that users can inspect different language
> results.
> >> If you require the correct language key, you should get pretty good
> >> retrieval speed.
> >>
> >> On Fri, Jan 20, 2012 at 3:35 AM, nibing <ni...@hotmail.com>
> > wrote:
> >>
> >> >
> >> > Hi, Jan Høydahl You are right. I am hoping to detect the language
> of
> > a
> >> > query, so that the serarching can be done according to the language
> >> > detected. Since people often type a few words, which is too few to
> > detect,
> >> > then it is hard to do that. Let me describe a little bit about the
> > solr
> >> > server in my design. It consists of several cores, corresponding to
> > the
> >> > several languages, which is built during indexing. Since language
> > detection
> >> > in indexing can be done with Tika identifier, then we are currently
> > OK. But
> >> > the problem is about searching. I want to do language detection first
> >> > before do searching in the individual cores. In the case that
> > detection
> >> > result is ambiguous and several languages are returned, we probably
> > returns
> >> > a set of results, and let user to decide which language set of
> results
> > they
> >> > want to look into. In general, it is just the same with the language
> >> > supported by google. Do you have some suggestions if I want to
> achieve
> >> > multilingual search described as above? Thank you.
> >> > Best Regards
> >> > Ni, Bing
> >> >
> >> > > Subject: Re: Tika0.10 language identifier in Solr3.5.0
> >> > > From: jan.asf@cominvent.com
> >> > > Date: Thu, 19 Jan 2012 12:31:01 +0100
> >> > > To: solr-user@lucene.apache.org
> >> > >
> >> > > Hi,
> >> > >
> >> > > You may use the string as you choose, for instance filtering
> >> > (fq=language_s:en) or for faceting (facet.field=language_s). What are
> > you
> >> > looking to do?
> >> > >
> >> > > What would you like to detect on the query side? The language of
> > the
> >> > search string? That is very hard since people type very few words
> into
> > the
> >> > search box.
> >> > >
> >> > > --
> >> > > Jan Høydahl, search solution architect
> >> > > Cominvent AS - www.cominvent.com
> >> > > Solr Training - www.solrtraining.com
> >> > >
> >> > > On 19. jan. 2012, at 09:22, nibing wrote:
> >> > >
> >> > > >
> >> > > > Hi, all,
> >> > > >
> >> > > >
> >> > > >
> >> > > > I am using Solr3.5.0 which applies Tika0.10 to do language
> > detection,
> >> > > > and I have a couple of questions about this function.
> >> > > >
> >> > > >
> >> > > >
> >> > > > 1. I can see the outcome of the language detection in a
> > field
> >> > > > "language_s". But what action will be taken
> > according to the different
> >> > > > language code? How to configure?
> >> > > >
> >> > > >
> >> > > >
> >> > > > 2. Currently the language detection only happens in
> > indexing. Is it
> >> > > > possible to use the function in searching as well? How to
> > configure?
> >> > > >
> >> > > >
> >> > > >
> >> > > > Many thanks.
> >> > > >
> >> > > >
> >> > > > Best Regards
> >> > > >
> >> > > > Ni, Bing
> >> > > >
> >> > >
> >> >
> >
>
Re: Tika0.10 language identifier in Solr3.5.0
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Ni, Bing
I believe you will need to pre-define fields for all languages you want to handle and specify an appropriate language-specific analyzer for each of those fields.
This also means that if you encounter a new language, you will need to adjust your schema to support a new language. Of course, your Lang Identifier will also need to be trained to recognize that language.
I know ElasticSearch allows you to specify a field analyzer at index time, which is very handy. I don't think you can do that with Solr today, but maybe I'm forgetting something! :)
Otis
----
Performance Monitoring SaaS for Solr - http://sematext.com/spm/solr-performance-monitoring/index.html
----- Original Message -----
> From: nibing <ni...@hotmail.com>
> To: solr-user@lucene.apache.org
> Cc:
> Sent: Friday, January 20, 2012 1:51 AM
> Subject: RE: Tika0.10 language identifier in Solr3.5.0
>
>
> Hi, Ted Dunning,
>
> Thank you for your reply. I can understand your point on putting a
> "language_s" field and then keeping all the files together, which
> speed-up searching.
>
> But then there occurs a problem of using analyzer in indexing. I assume files
> encoded in different language should be handled using different analyzers, i.e.
> different tokenizers and filters. Can you elaborate a little bit on the design
> that you propose, especially in how files encoded in different languages can be
> handled during the indexing?
> Thank you.
>
> Best Regards
>
> Ni, Bing
>
>
>> From: ted.dunning@gmail.com
>> Date: Fri, 20 Jan 2012 03:55:48 +0000
>> Subject: Re: Tika0.10 language identifier in Solr3.5.0
>> To: solr-user@lucene.apache.org
>>
>> Normally this is done by putting a field on each document rather than
>> separating the documents into separate corpora. Keeping them together
>> makes the final search faster.
>>
>> At query time, you can add all of the language keys that you think are
>> relevant based on your language id applied to the query and then group the
>> results on those keys so that users can inspect different language results.
>> If you require the correct language key, you should get pretty good
>> retrieval speed.
>>
>> On Fri, Jan 20, 2012 at 3:35 AM, nibing <ni...@hotmail.com>
> wrote:
>>
>> >
>> > Hi, Jan Høydahl You are right. I am hoping to detect the language of
> a
>> > query, so that the serarching can be done according to the language
>> > detected. Since people often type a few words, which is too few to
> detect,
>> > then it is hard to do that. Let me describe a little bit about the
> solr
>> > server in my design. It consists of several cores, corresponding to
> the
>> > several languages, which is built during indexing. Since language
> detection
>> > in indexing can be done with Tika identifier, then we are currently
> OK. But
>> > the problem is about searching. I want to do language detection first
>> > before do searching in the individual cores. In the case that
> detection
>> > result is ambiguous and several languages are returned, we probably
> returns
>> > a set of results, and let user to decide which language set of results
> they
>> > want to look into. In general, it is just the same with the language
>> > supported by google. Do you have some suggestions if I want to achieve
>> > multilingual search described as above? Thank you.
>> > Best Regards
>> > Ni, Bing
>> >
>> > > Subject: Re: Tika0.10 language identifier in Solr3.5.0
>> > > From: jan.asf@cominvent.com
>> > > Date: Thu, 19 Jan 2012 12:31:01 +0100
>> > > To: solr-user@lucene.apache.org
>> > >
>> > > Hi,
>> > >
>> > > You may use the string as you choose, for instance filtering
>> > (fq=language_s:en) or for faceting (facet.field=language_s). What are
> you
>> > looking to do?
>> > >
>> > > What would you like to detect on the query side? The language of
> the
>> > search string? That is very hard since people type very few words into
> the
>> > search box.
>> > >
>> > > --
>> > > Jan Høydahl, search solution architect
>> > > Cominvent AS - www.cominvent.com
>> > > Solr Training - www.solrtraining.com
>> > >
>> > > On 19. jan. 2012, at 09:22, nibing wrote:
>> > >
>> > > >
>> > > > Hi, all,
>> > > >
>> > > >
>> > > >
>> > > > I am using Solr3.5.0 which applies Tika0.10 to do language
> detection,
>> > > > and I have a couple of questions about this function.
>> > > >
>> > > >
>> > > >
>> > > > 1. I can see the outcome of the language detection in a
> field
>> > > > "language_s". But what action will be taken
> according to the different
>> > > > language code? How to configure?
>> > > >
>> > > >
>> > > >
>> > > > 2. Currently the language detection only happens in
> indexing. Is it
>> > > > possible to use the function in searching as well? How to
> configure?
>> > > >
>> > > >
>> > > >
>> > > > Many thanks.
>> > > >
>> > > >
>> > > > Best Regards
>> > > >
>> > > > Ni, Bing
>> > > >
>> > >
>> >
>
RE: Tika0.10 language identifier in Solr3.5.0
Posted by nibing <ni...@hotmail.com>.
Hi, Ted Dunning,
Thank you for your reply. I can understand your point on putting a "language_s" field and then keeping all the files together, which speed-up searching.
But then there occurs a problem of using analyzer in indexing. I assume files encoded in different language should be handled using different analyzers, i.e. different tokenizers and filters. Can you elaborate a little bit on the design that you propose, especially in how files encoded in different languages can be handled during the indexing?
Thank you.
Best Regards
Ni, Bing
> From: ted.dunning@gmail.com
> Date: Fri, 20 Jan 2012 03:55:48 +0000
> Subject: Re: Tika0.10 language identifier in Solr3.5.0
> To: solr-user@lucene.apache.org
>
> Normally this is done by putting a field on each document rather than
> separating the documents into separate corpora. Keeping them together
> makes the final search faster.
>
> At query time, you can add all of the language keys that you think are
> relevant based on your language id applied to the query and then group the
> results on those keys so that users can inspect different language results.
> If you require the correct language key, you should get pretty good
> retrieval speed.
>
> On Fri, Jan 20, 2012 at 3:35 AM, nibing <ni...@hotmail.com> wrote:
>
> >
> > Hi, Jan Høydahl You are right. I am hoping to detect the language of a
> > query, so that the serarching can be done according to the language
> > detected. Since people often type a few words, which is too few to detect,
> > then it is hard to do that. Let me describe a little bit about the solr
> > server in my design. It consists of several cores, corresponding to the
> > several languages, which is built during indexing. Since language detection
> > in indexing can be done with Tika identifier, then we are currently OK. But
> > the problem is about searching. I want to do language detection first
> > before do searching in the individual cores. In the case that detection
> > result is ambiguous and several languages are returned, we probably returns
> > a set of results, and let user to decide which language set of results they
> > want to look into. In general, it is just the same with the language
> > supported by google. Do you have some suggestions if I want to achieve
> > multilingual search described as above? Thank you.
> > Best Regards
> > Ni, Bing
> >
> > > Subject: Re: Tika0.10 language identifier in Solr3.5.0
> > > From: jan.asf@cominvent.com
> > > Date: Thu, 19 Jan 2012 12:31:01 +0100
> > > To: solr-user@lucene.apache.org
> > >
> > > Hi,
> > >
> > > You may use the string as you choose, for instance filtering
> > (fq=language_s:en) or for faceting (facet.field=language_s). What are you
> > looking to do?
> > >
> > > What would you like to detect on the query side? The language of the
> > search string? That is very hard since people type very few words into the
> > search box.
> > >
> > > --
> > > Jan Høydahl, search solution architect
> > > Cominvent AS - www.cominvent.com
> > > Solr Training - www.solrtraining.com
> > >
> > > On 19. jan. 2012, at 09:22, nibing wrote:
> > >
> > > >
> > > > Hi, all,
> > > >
> > > >
> > > >
> > > > I am using Solr3.5.0 which applies Tika0.10 to do language detection,
> > > > and I have a couple of questions about this function.
> > > >
> > > >
> > > >
> > > > 1. I can see the outcome of the language detection in a field
> > > > "language_s". But what action will be taken according to the different
> > > > language code? How to configure?
> > > >
> > > >
> > > >
> > > > 2. Currently the language detection only happens in indexing. Is it
> > > > possible to use the function in searching as well? How to configure?
> > > >
> > > >
> > > >
> > > > Many thanks.
> > > >
> > > >
> > > > Best Regards
> > > >
> > > > Ni, Bing
> > > >
> > >
> >
Re: Tika0.10 language identifier in Solr3.5.0
Posted by Ted Dunning <te...@gmail.com>.
Normally this is done by putting a field on each document rather than
separating the documents into separate corpora. Keeping them together
makes the final search faster.
At query time, you can add all of the language keys that you think are
relevant based on your language id applied to the query and then group the
results on those keys so that users can inspect different language results.
If you require the correct language key, you should get pretty good
retrieval speed.
On Fri, Jan 20, 2012 at 3:35 AM, nibing <ni...@hotmail.com> wrote:
>
> Hi, Jan Høydahl You are right. I am hoping to detect the language of a
> query, so that the serarching can be done according to the language
> detected. Since people often type a few words, which is too few to detect,
> then it is hard to do that. Let me describe a little bit about the solr
> server in my design. It consists of several cores, corresponding to the
> several languages, which is built during indexing. Since language detection
> in indexing can be done with Tika identifier, then we are currently OK. But
> the problem is about searching. I want to do language detection first
> before do searching in the individual cores. In the case that detection
> result is ambiguous and several languages are returned, we probably returns
> a set of results, and let user to decide which language set of results they
> want to look into. In general, it is just the same with the language
> supported by google. Do you have some suggestions if I want to achieve
> multilingual search described as above? Thank you.
> Best Regards
> Ni, Bing
>
> > Subject: Re: Tika0.10 language identifier in Solr3.5.0
> > From: jan.asf@cominvent.com
> > Date: Thu, 19 Jan 2012 12:31:01 +0100
> > To: solr-user@lucene.apache.org
> >
> > Hi,
> >
> > You may use the string as you choose, for instance filtering
> (fq=language_s:en) or for faceting (facet.field=language_s). What are you
> looking to do?
> >
> > What would you like to detect on the query side? The language of the
> search string? That is very hard since people type very few words into the
> search box.
> >
> > --
> > Jan Høydahl, search solution architect
> > Cominvent AS - www.cominvent.com
> > Solr Training - www.solrtraining.com
> >
> > On 19. jan. 2012, at 09:22, nibing wrote:
> >
> > >
> > > Hi, all,
> > >
> > >
> > >
> > > I am using Solr3.5.0 which applies Tika0.10 to do language detection,
> > > and I have a couple of questions about this function.
> > >
> > >
> > >
> > > 1. I can see the outcome of the language detection in a field
> > > "language_s". But what action will be taken according to the different
> > > language code? How to configure?
> > >
> > >
> > >
> > > 2. Currently the language detection only happens in indexing. Is it
> > > possible to use the function in searching as well? How to configure?
> > >
> > >
> > >
> > > Many thanks.
> > >
> > >
> > > Best Regards
> > >
> > > Ni, Bing
> > >
> >
>
RE: Tika0.10 language identifier in Solr3.5.0
Posted by nibing <ni...@hotmail.com>.
Hi, Jan Høydahl You are right. I am hoping to detect the language of a query, so that the serarching can be done according to the language detected. Since people often type a few words, which is too few to detect, then it is hard to do that. Let me describe a little bit about the solr server in my design. It consists of several cores, corresponding to the several languages, which is built during indexing. Since language detection in indexing can be done with Tika identifier, then we are currently OK. But the problem is about searching. I want to do language detection first before do searching in the individual cores. In the case that detection result is ambiguous and several languages are returned, we probably returns a set of results, and let user to decide which language set of results they want to look into. In general, it is just the same with the language supported by google. Do you have some suggestions if I want to achieve multilingual search described as above? Thank you.
Best Regards
Ni, Bing
> Subject: Re: Tika0.10 language identifier in Solr3.5.0
> From: jan.asf@cominvent.com
> Date: Thu, 19 Jan 2012 12:31:01 +0100
> To: solr-user@lucene.apache.org
>
> Hi,
>
> You may use the string as you choose, for instance filtering (fq=language_s:en) or for faceting (facet.field=language_s). What are you looking to do?
>
> What would you like to detect on the query side? The language of the search string? That is very hard since people type very few words into the search box.
>
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.cominvent.com
> Solr Training - www.solrtraining.com
>
> On 19. jan. 2012, at 09:22, nibing wrote:
>
> >
> > Hi, all,
> >
> >
> >
> > I am using Solr3.5.0 which applies Tika0.10 to do language detection,
> > and I have a couple of questions about this function.
> >
> >
> >
> > 1. I can see the outcome of the language detection in a field
> > "language_s". But what action will be taken according to the different
> > language code? How to configure?
> >
> >
> >
> > 2. Currently the language detection only happens in indexing. Is it
> > possible to use the function in searching as well? How to configure?
> >
> >
> >
> > Many thanks.
> >
> >
> > Best Regards
> >
> > Ni, Bing
> >
>
Re: Tika0.10 language identifier in Solr3.5.0
Posted by Jan Høydahl <ja...@cominvent.com>.
Hi,
You may use the string as you choose, for instance filtering (fq=language_s:en) or for faceting (facet.field=language_s). What are you looking to do?
What would you like to detect on the query side? The language of the search string? That is very hard since people type very few words into the search box.
--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com
Solr Training - www.solrtraining.com
On 19. jan. 2012, at 09:22, nibing wrote:
>
> Hi, all,
>
>
>
> I am using Solr3.5.0 which applies Tika0.10 to do language detection,
> and I have a couple of questions about this function.
>
>
>
> 1. I can see the outcome of the language detection in a field
> "language_s". But what action will be taken according to the different
> language code? How to configure?
>
>
>
> 2. Currently the language detection only happens in indexing. Is it
> possible to use the function in searching as well? How to configure?
>
>
>
> Many thanks.
>
>
> Best Regards
>
> Ni, Bing
>
Re: Tika0.10 language identifier in Solr3.5.0
Posted by Alessio Crisantemi <al...@gioconews.it>.
Dear all,
I can I do for indexing a complete directory with many pdf files on Solr?
Alessio Crisantemi
Direttore Responsabile Gioconews.it
www.gioconews.it
t: (+39)0744461296
f: (+39)0744461362
bb: (+39)3477939054
e: alessio.crisantemi@gioconews.it
-----------------------------------------------------------------------------------------------------------------------------------
Ai sensi del D.Lgs. 196/2003 si precisa che le informazioni contenute in
questo messaggio
sono riservate ed a uso esclusivo del destinatario. Pertanto, qualora il
messaggio sopra citato Le fosse pervenuto per errore, La invitiamo a
provvedere quanto prima alla sua eliminazione senza copiarlo e/o inoltrarlo
a terzi, dandocene gentilmente comunicazione.
Pursuant to Legislative Decree No. 196/2003, you are hereby informed that
this message
contains confidential information intended only for the use of the
addressee. If you are not the addressee, and have received this message by
mistake, please delete it and immediately notify us. You may not copy or
disseminate this message to anyone.
-----------------------------------------------------------------------------------------------------------------------------------
-----Messaggio originale-----
From: nibing
Sent: Thursday, January 19, 2012 9:22 AM
To: solr-user@lucene.apache.org
Subject: Tika0.10 language identifier in Solr3.5.0
Hi, all,
I am using Solr3.5.0 which applies Tika0.10 to do language detection,
and I have a couple of questions about this function.
1. I can see the outcome of the language detection in a field
"language_s". But what action will be taken according to the different
language code? How to configure?
2. Currently the language detection only happens in indexing. Is it
possible to use the function in searching as well? How to configure?
Many thanks.
Best Regards
Ni, Bing