[jira] [Commented] (SOLR-6492) Solr field type that supports multiple, dynamic analyzers

["Jan Rasehorn (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SOLR-6492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16028252#comment-16028252 ] 

Jan Rasehorn commented on SOLR-6492:
------------------------------------

Hi Guys, this sounds like a solution for indexing a whole document when the document language is known upfront. 
But what if the language is not known upfront or if a document contains different text paragraphs with possibly different languages - like it can often be found in support tickets?

Since I did not like the approach using separate fields, I did it the following way:
1. I wrote a tokenizer that detects the paragraphs based on a given regexp (a result of cleaning up the support ticket text)
2. The tokenizer detects the paragraph language at runtime (using the solr built in language detector)
3. The tokenizer runs part Open NLP POS tagging depending on the language it identified and saves the POS tags in the type attribute for each token. 
    The language is stored as payload for each token.
4. I developed a "Delegating filter", which only delegates the "incrementToken" call to the filter (stemmer) if the payload value matched the filter value. This way I can configure in schema.xml, which stemmer to use for which language.

With this approach I do not depend on knowning the document language upfront.
What do you think?


> Solr field type that supports multiple, dynamic analyzers
> ---------------------------------------------------------
>
>                 Key: SOLR-6492
>                 URL: https://issues.apache.org/jira/browse/SOLR-6492
>             Project: Solr
>          Issue Type: New Feature
>          Components: Schema and Analysis
>            Reporter: Trey Grainger
>             Fix For: 5.0
>
>
> A common request - particularly for multilingual search - is to be able to support one or more dynamically-selected analyzers for a field. For example, someone may have a "content" field and pass in a document in Greek (using an Analyzer with Tokenizer/Filters for German), a separate document in English (using an English Analyzer), and possibly even a field with mixed-language content in Greek and English. This latter case could pass the content separately through both an analyzer defined for Greek and another Analyzer defined for English, stacking or concatenating the token streams based upon the use-case.
> There are some distinct advantages in terms of index size and query performance which can be obtained by stacking terms from multiple analyzers in the same field instead of duplicating content in separate fields and searching across multiple fields. 
> Other non-multilingual use cases may include things like switching to a different analyzer for the same field to remove a feature (i.e. turning on/off query-time synonyms against the same field on a per-query basis).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org