[jira] [Commented] (NUTCH-923) Multilingual support for Solr-index-mapping

["Luca Cavanna (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NUTCH-923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13427944#comment-13427944 ] 

Luca Cavanna commented on NUTCH-923:
------------------------------------

Good to know about the language detection mechanism in Solr. But I think this nutch feature would be useful anyway. There are cases where you are crawling multilanguage pages and you know their language based on a url pattern, you don't even need to auto-detect it. You could then use the subcollection plugin to store the language in a field, but then you can't use stemming unless you enable the Solr language identification, which auto-detects the language that I already know in advance while crawling and indexes content in a different field based on the detected language. The solr mapping abilities should be made a little more flexible I guess.
                
> Multilingual support for Solr-index-mapping
> -------------------------------------------
>
>                 Key: NUTCH-923
>                 URL: https://issues.apache.org/jira/browse/NUTCH-923
>             Project: Nutch
>          Issue Type: Improvement
>          Components: indexer
>    Affects Versions: 1.2
>            Reporter: Matthias Agethle
>            Assignee: Markus Jelsma
>            Priority: Minor
>         Attachments: patch-923-nutch-release-1.2.txt
>
>
> It would be useful to extend the mapping-possibilites when indexing to solr.
> One useful feature would be to use the detected language of the html page (for example via the language-identifier plugin) and send the content to corresponding language-aware solr-fields.
> The mapping file could be as follows:
> <field dest="lang" source="lang"/>
> <field dest="title_${lang}" source="title" />
> so that the title-field gets mapped to title_en for English-pages and tilte_fr for French pages.
> What do you think? Could this be useful also to others?
> Or are there already other solutions out there?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira