Multiple Languages in Same Core

[Jeremy Thomerson <je...@thomersonfamily.com>]

I recently deployed Solr to back the site search feature of a site I work
on. The site itself is available in hundreds of languages. With the initial
release of site search we have enabled the feature for ten of those
languages. This is distributed across eight cores, with two Chinese
languages plus Korean combined into one CJK core and each of the other
seven languages in their own individual cores. The reason for splitting
these into separate cores was so that we could have the same field names
across all cores but have different configuration for analyzers, etc, per
core.

Now I have some questions on this approach.

1) Scalability: Considering I need to scale this to many dozens more
languages, perhaps hundreds more, is there a better way so that I don't end
up needing dozens or hundreds of cores? My initial plan was that many
languages that didn't have special support within Solr would simply get
lumped into a single "default" core that has some default analyzers that
are applicable to the majority of languages.

1b) Related to this: is there a practical limit to the number of cores that
can be run on one instance of Lucene?

2) Auto Suggest: In phase two I intend to add auto-suggestions as a user
types a query. In reviewing how this is implemented and how the suggestion
dictionary is built I have concerns. If I have more than one language in a
single core (and I keep the same field name for suggestions on all
languages within a core) then it seems that I could get suggestions from
another language returned with a suggest query. Is there a way to build a
separate dictionary for each language, but keep these languages within the
same core?

If it's helpful to know: I have a field in every core for "Locale". Values
will be the locale of the language of that document, i.e. "en", "es",
"zh_hans", etc. I'd like to be able to: 1) when building a suggestion
dictionary, divide it into multiple dictionaries, grouping them by locale,
and 2) supply a parameter to the suggest query that allows the suggest
component to only return suggestions from the appropriate dictionary for
that locale.

If the answer to #1 is "keep splitting groups of languages that have
different analyzers into their own cores" and the answer to #2 is "that's
not supported", then I'd be curious: where would I start to write my own
extension that supported #2? I looked last night at the suggest lookup
classes, dictionary classes, etc. But I didn't see a clear point where it
would be clean to implement something like I'm suggesting above.

Best Regards,
Jeremy Thomerson

Re: Multiple Languages in Same Core

[Jeremy Thomerson <je...@thomersonfamily.com>]

Thanks Trey! Last week I ordered the eBook. I look forward to seeing the
information in it.

Jeremy


On Thu, Mar 27, 2014 at 6:03 PM, Trey Grainger <so...@gmail.com> wrote:

> In addition to the two approaches Liu Bo mentioned (separate core per
> language and separate field per language), it is also possible to put
> multiple languages in a single field. This saves you the overhead of
> multiple cores and of having to search across multiple fields at query
> time. The idea here is that you can run multiple analyzers (i.e. one for
> German, one for English, one for Chinese, etc.) and stack the outputted
> TokenStreams for each of these within a single field. It is also possible
> to swap out the languages you want to use on a case-by-case basis (i.e.
> per-document, per field, or even per word) if you really need to for
> advanced use cases.
>
> All three of these methods, including code examples and the pros and cons
> of each are discussed in the Multilingual Search chapter of Solr in Action,
> which Alexandre referenced. If you don't have the book, you can also just
> download and run the code examples for free, though they may be harder to
> follow without the context from the book.
>
> Thanks,
>
> Trey Grainger
> Co-author, Solr in Action
> Director of Engineering, Search & Analytics @CareerBuilder
>
>
>
>
>
> On Wed, Mar 26, 2014 at 4:34 AM, Liu Bo <di...@gmail.com> wrote:
>
> > Hi Jeremy
> >
> > There're a lot of multi language discussions, two main approaches
> >  1. like yours, a language is one core
> >  2. all in one core, different language has it's own field.
> >
> > We have multi-language support in a single core, each multilingual field
> > has it's own suffix such as name_en_US. We customized query handler to
> hide
> > the query details to client.
> > The main reason we want to do this is about NRT index and search,
> > take product for example:
> >
> >     product has price, quantity which is common and it's used by
> filtering
> > and sorting, name, description is multi language field,
> >     if we split product in do different cores, the common field updating
> > may end up a update in all of the multi language cores.
> >
> > As to scalability, we don't change solr cores/collections when a new
> > language is added, but we probably need update our customized index
> process
> > and run a full re-index.
> >
> > This approach suits our requirement for now, but you may have your own
> > concerns.
> >
> > We have similar "suggest filter" problem like yours, we want to return
> > suggest result filtering by stores. I can't find a way to build
> dictionary
> > with query at my version of solr 4.6
> >
> > What I do is run a query on a N-Gram analyzed field and with filter
> queries
> > on store_id field. The "suggest" is actually a query. It may not perform
> as
> > well as suggestion but can do the trick.
> >
> > You can try it to build a additional N-GRAM field for suggestion only and
> > search on it with fq on your "Locale" field.
> >
> > All the best
> >
> > Liu Bo
> >
> >
> >
> >
> > On 25 March 2014 09:15, Alexandre Rafalovitch <ar...@gmail.com>
> wrote:
> >
> > > Solr In Action has a significant discussion on the multi-lingual
> > > approach. They also have some code samples out there. Might be worth a
> > > look
> > >
> > > Regards,
> > >    Alex.
> > > Personal website: http://www.outerthoughts.com/
> > > LinkedIn: http://www.linkedin.com/in/alexandrerafalovitch
> > > - Time is the quality of nature that keeps events from happening all
> > > at once. Lately, it doesn't seem to be working.  (Anonymous  - via GTD
> > > book)
> > >
> > >
> > > On Tue, Mar 25, 2014 at 4:43 AM, Jeremy Thomerson
> > > <je...@thomersonfamily.com> wrote:
> > > > I recently deployed Solr to back the site search feature of a site I
> > work
> > > > on. The site itself is available in hundreds of languages. With the
> > > initial
> > > > release of site search we have enabled the feature for ten of those
> > > > languages. This is distributed across eight cores, with two Chinese
> > > > languages plus Korean combined into one CJK core and each of the
> other
> > > > seven languages in their own individual cores. The reason for
> splitting
> > > > these into separate cores was so that we could have the same field
> > names
> > > > across all cores but have different configuration for analyzers, etc,
> > per
> > > > core.
> > > >
> > > > Now I have some questions on this approach.
> > > >
> > > > 1) Scalability: Considering I need to scale this to many dozens more
> > > > languages, perhaps hundreds more, is there a better way so that I
> don't
> > > end
> > > > up needing dozens or hundreds of cores? My initial plan was that many
> > > > languages that didn't have special support within Solr would simply
> get
> > > > lumped into a single "default" core that has some default analyzers
> > that
> > > > are applicable to the majority of languages.
> > > >
> > > > 1b) Related to this: is there a practical limit to the number of
> cores
> > > that
> > > > can be run on one instance of Lucene?
> > > >
> > > > 2) Auto Suggest: In phase two I intend to add auto-suggestions as a
> > user
> > > > types a query. In reviewing how this is implemented and how the
> > > suggestion
> > > > dictionary is built I have concerns. If I have more than one language
> > in
> > > a
> > > > single core (and I keep the same field name for suggestions on all
> > > > languages within a core) then it seems that I could get suggestions
> > from
> > > > another language returned with a suggest query. Is there a way to
> > build a
> > > > separate dictionary for each language, but keep these languages
> within
> > > the
> > > > same core?
> > > >
> > > > If it's helpful to know: I have a field in every core for "Locale".
> > > Values
> > > > will be the locale of the language of that document, i.e. "en", "es",
> > > > "zh_hans", etc. I'd like to be able to: 1) when building a suggestion
> > > > dictionary, divide it into multiple dictionaries, grouping them by
> > > locale,
> > > > and 2) supply a parameter to the suggest query that allows the
> suggest
> > > > component to only return suggestions from the appropriate dictionary
> > for
> > > > that locale.
> > > >
> > > > If the answer to #1 is "keep splitting groups of languages that have
> > > > different analyzers into their own cores" and the answer to #2 is
> > "that's
> > > > not supported", then I'd be curious: where would I start to write my
> > own
> > > > extension that supported #2? I looked last night at the suggest
> lookup
> > > > classes, dictionary classes, etc. But I didn't see a clear point
> where
> > it
> > > > would be clean to implement something like I'm suggesting above.
> > > >
> > > > Best Regards,
> > > > Jeremy Thomerson
> > >
> >
> >
> >
> > --
> > All the best
> >
> > Liu Bo
> >
>

Re: Multiple Languages in Same Core

[Trey Grainger <so...@gmail.com>]

In addition to the two approaches Liu Bo mentioned (separate core per
language and separate field per language), it is also possible to put
multiple languages in a single field. This saves you the overhead of
multiple cores and of having to search across multiple fields at query
time. The idea here is that you can run multiple analyzers (i.e. one for
German, one for English, one for Chinese, etc.) and stack the outputted
TokenStreams for each of these within a single field. It is also possible
to swap out the languages you want to use on a case-by-case basis (i.e.
per-document, per field, or even per word) if you really need to for
advanced use cases.

All three of these methods, including code examples and the pros and cons
of each are discussed in the Multilingual Search chapter of Solr in Action,
which Alexandre referenced. If you don't have the book, you can also just
download and run the code examples for free, though they may be harder to
follow without the context from the book.

Thanks,

Trey Grainger
Co-author, Solr in Action
Director of Engineering, Search & Analytics @CareerBuilder





On Wed, Mar 26, 2014 at 4:34 AM, Liu Bo <di...@gmail.com> wrote:

> Hi Jeremy
>
> There're a lot of multi language discussions, two main approaches
>  1. like yours, a language is one core
>  2. all in one core, different language has it's own field.
>
> We have multi-language support in a single core, each multilingual field
> has it's own suffix such as name_en_US. We customized query handler to hide
> the query details to client.
> The main reason we want to do this is about NRT index and search,
> take product for example:
>
>     product has price, quantity which is common and it's used by filtering
> and sorting, name, description is multi language field,
>     if we split product in do different cores, the common field updating
> may end up a update in all of the multi language cores.
>
> As to scalability, we don't change solr cores/collections when a new
> language is added, but we probably need update our customized index process
> and run a full re-index.
>
> This approach suits our requirement for now, but you may have your own
> concerns.
>
> We have similar "suggest filter" problem like yours, we want to return
> suggest result filtering by stores. I can't find a way to build dictionary
> with query at my version of solr 4.6
>
> What I do is run a query on a N-Gram analyzed field and with filter queries
> on store_id field. The "suggest" is actually a query. It may not perform as
> well as suggestion but can do the trick.
>
> You can try it to build a additional N-GRAM field for suggestion only and
> search on it with fq on your "Locale" field.
>
> All the best
>
> Liu Bo
>
>
>
>
> On 25 March 2014 09:15, Alexandre Rafalovitch <ar...@gmail.com> wrote:
>
> > Solr In Action has a significant discussion on the multi-lingual
> > approach. They also have some code samples out there. Might be worth a
> > look
> >
> > Regards,
> >    Alex.
> > Personal website: http://www.outerthoughts.com/
> > LinkedIn: http://www.linkedin.com/in/alexandrerafalovitch
> > - Time is the quality of nature that keeps events from happening all
> > at once. Lately, it doesn't seem to be working.  (Anonymous  - via GTD
> > book)
> >
> >
> > On Tue, Mar 25, 2014 at 4:43 AM, Jeremy Thomerson
> > <je...@thomersonfamily.com> wrote:
> > > I recently deployed Solr to back the site search feature of a site I
> work
> > > on. The site itself is available in hundreds of languages. With the
> > initial
> > > release of site search we have enabled the feature for ten of those
> > > languages. This is distributed across eight cores, with two Chinese
> > > languages plus Korean combined into one CJK core and each of the other
> > > seven languages in their own individual cores. The reason for splitting
> > > these into separate cores was so that we could have the same field
> names
> > > across all cores but have different configuration for analyzers, etc,
> per
> > > core.
> > >
> > > Now I have some questions on this approach.
> > >
> > > 1) Scalability: Considering I need to scale this to many dozens more
> > > languages, perhaps hundreds more, is there a better way so that I don't
> > end
> > > up needing dozens or hundreds of cores? My initial plan was that many
> > > languages that didn't have special support within Solr would simply get
> > > lumped into a single "default" core that has some default analyzers
> that
> > > are applicable to the majority of languages.
> > >
> > > 1b) Related to this: is there a practical limit to the number of cores
> > that
> > > can be run on one instance of Lucene?
> > >
> > > 2) Auto Suggest: In phase two I intend to add auto-suggestions as a
> user
> > > types a query. In reviewing how this is implemented and how the
> > suggestion
> > > dictionary is built I have concerns. If I have more than one language
> in
> > a
> > > single core (and I keep the same field name for suggestions on all
> > > languages within a core) then it seems that I could get suggestions
> from
> > > another language returned with a suggest query. Is there a way to
> build a
> > > separate dictionary for each language, but keep these languages within
> > the
> > > same core?
> > >
> > > If it's helpful to know: I have a field in every core for "Locale".
> > Values
> > > will be the locale of the language of that document, i.e. "en", "es",
> > > "zh_hans", etc. I'd like to be able to: 1) when building a suggestion
> > > dictionary, divide it into multiple dictionaries, grouping them by
> > locale,
> > > and 2) supply a parameter to the suggest query that allows the suggest
> > > component to only return suggestions from the appropriate dictionary
> for
> > > that locale.
> > >
> > > If the answer to #1 is "keep splitting groups of languages that have
> > > different analyzers into their own cores" and the answer to #2 is
> "that's
> > > not supported", then I'd be curious: where would I start to write my
> own
> > > extension that supported #2? I looked last night at the suggest lookup
> > > classes, dictionary classes, etc. But I didn't see a clear point where
> it
> > > would be clean to implement something like I'm suggesting above.
> > >
> > > Best Regards,
> > > Jeremy Thomerson
> >
>
>
>
> --
> All the best
>
> Liu Bo
>

Re: Multiple Languages in Same Core

[Liu Bo <di...@gmail.com>]

Hi Jeremy

There're a lot of multi language discussions, two main approaches
 1. like yours, a language is one core
 2. all in one core, different language has it's own field.

We have multi-language support in a single core, each multilingual field
has it's own suffix such as name_en_US. We customized query handler to hide
the query details to client.
The main reason we want to do this is about NRT index and search,
take product for example:

    product has price, quantity which is common and it's used by filtering
and sorting, name, description is multi language field,
    if we split product in do different cores, the common field updating
may end up a update in all of the multi language cores.

As to scalability, we don't change solr cores/collections when a new
language is added, but we probably need update our customized index process
and run a full re-index.

This approach suits our requirement for now, but you may have your own
concerns.

We have similar "suggest filter" problem like yours, we want to return
suggest result filtering by stores. I can't find a way to build dictionary
with query at my version of solr 4.6

What I do is run a query on a N-Gram analyzed field and with filter queries
on store_id field. The "suggest" is actually a query. It may not perform as
well as suggestion but can do the trick.

You can try it to build a additional N-GRAM field for suggestion only and
search on it with fq on your "Locale" field.

All the best

Liu Bo




On 25 March 2014 09:15, Alexandre Rafalovitch <ar...@gmail.com> wrote:

> Solr In Action has a significant discussion on the multi-lingual
> approach. They also have some code samples out there. Might be worth a
> look
>
> Regards,
>    Alex.
> Personal website: http://www.outerthoughts.com/
> LinkedIn: http://www.linkedin.com/in/alexandrerafalovitch
> - Time is the quality of nature that keeps events from happening all
> at once. Lately, it doesn't seem to be working.  (Anonymous  - via GTD
> book)
>
>
> On Tue, Mar 25, 2014 at 4:43 AM, Jeremy Thomerson
> <je...@thomersonfamily.com> wrote:
> > I recently deployed Solr to back the site search feature of a site I work
> > on. The site itself is available in hundreds of languages. With the
> initial
> > release of site search we have enabled the feature for ten of those
> > languages. This is distributed across eight cores, with two Chinese
> > languages plus Korean combined into one CJK core and each of the other
> > seven languages in their own individual cores. The reason for splitting
> > these into separate cores was so that we could have the same field names
> > across all cores but have different configuration for analyzers, etc, per
> > core.
> >
> > Now I have some questions on this approach.
> >
> > 1) Scalability: Considering I need to scale this to many dozens more
> > languages, perhaps hundreds more, is there a better way so that I don't
> end
> > up needing dozens or hundreds of cores? My initial plan was that many
> > languages that didn't have special support within Solr would simply get
> > lumped into a single "default" core that has some default analyzers that
> > are applicable to the majority of languages.
> >
> > 1b) Related to this: is there a practical limit to the number of cores
> that
> > can be run on one instance of Lucene?
> >
> > 2) Auto Suggest: In phase two I intend to add auto-suggestions as a user
> > types a query. In reviewing how this is implemented and how the
> suggestion
> > dictionary is built I have concerns. If I have more than one language in
> a
> > single core (and I keep the same field name for suggestions on all
> > languages within a core) then it seems that I could get suggestions from
> > another language returned with a suggest query. Is there a way to build a
> > separate dictionary for each language, but keep these languages within
> the
> > same core?
> >
> > If it's helpful to know: I have a field in every core for "Locale".
> Values
> > will be the locale of the language of that document, i.e. "en", "es",
> > "zh_hans", etc. I'd like to be able to: 1) when building a suggestion
> > dictionary, divide it into multiple dictionaries, grouping them by
> locale,
> > and 2) supply a parameter to the suggest query that allows the suggest
> > component to only return suggestions from the appropriate dictionary for
> > that locale.
> >
> > If the answer to #1 is "keep splitting groups of languages that have
> > different analyzers into their own cores" and the answer to #2 is "that's
> > not supported", then I'd be curious: where would I start to write my own
> > extension that supported #2? I looked last night at the suggest lookup
> > classes, dictionary classes, etc. But I didn't see a clear point where it
> > would be clean to implement something like I'm suggesting above.
> >
> > Best Regards,
> > Jeremy Thomerson
>



-- 
All the best

Liu Bo

Re: Multiple Languages in Same Core

[Alexandre Rafalovitch <ar...@gmail.com>]

Solr In Action has a significant discussion on the multi-lingual
approach. They also have some code samples out there. Might be worth a
look

Regards,
   Alex.
Personal website: http://www.outerthoughts.com/
LinkedIn: http://www.linkedin.com/in/alexandrerafalovitch
- Time is the quality of nature that keeps events from happening all
at once. Lately, it doesn't seem to be working.  (Anonymous  - via GTD
book)


On Tue, Mar 25, 2014 at 4:43 AM, Jeremy Thomerson
<je...@thomersonfamily.com> wrote:
> I recently deployed Solr to back the site search feature of a site I work
> on. The site itself is available in hundreds of languages. With the initial
> release of site search we have enabled the feature for ten of those
> languages. This is distributed across eight cores, with two Chinese
> languages plus Korean combined into one CJK core and each of the other
> seven languages in their own individual cores. The reason for splitting
> these into separate cores was so that we could have the same field names
> across all cores but have different configuration for analyzers, etc, per
> core.
>
> Now I have some questions on this approach.
>
> 1) Scalability: Considering I need to scale this to many dozens more
> languages, perhaps hundreds more, is there a better way so that I don't end
> up needing dozens or hundreds of cores? My initial plan was that many
> languages that didn't have special support within Solr would simply get
> lumped into a single "default" core that has some default analyzers that
> are applicable to the majority of languages.
>
> 1b) Related to this: is there a practical limit to the number of cores that
> can be run on one instance of Lucene?
>
> 2) Auto Suggest: In phase two I intend to add auto-suggestions as a user
> types a query. In reviewing how this is implemented and how the suggestion
> dictionary is built I have concerns. If I have more than one language in a
> single core (and I keep the same field name for suggestions on all
> languages within a core) then it seems that I could get suggestions from
> another language returned with a suggest query. Is there a way to build a
> separate dictionary for each language, but keep these languages within the
> same core?
>
> If it's helpful to know: I have a field in every core for "Locale". Values
> will be the locale of the language of that document, i.e. "en", "es",
> "zh_hans", etc. I'd like to be able to: 1) when building a suggestion
> dictionary, divide it into multiple dictionaries, grouping them by locale,
> and 2) supply a parameter to the suggest query that allows the suggest
> component to only return suggestions from the appropriate dictionary for
> that locale.
>
> If the answer to #1 is "keep splitting groups of languages that have
> different analyzers into their own cores" and the answer to #2 is "that's
> not supported", then I'd be curious: where would I start to write my own
> extension that supported #2? I looked last night at the suggest lookup
> classes, dictionary classes, etc. But I didn't see a clear point where it
> would be clean to implement something like I'm suggesting above.
>
> Best Regards,
> Jeremy Thomerson