You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Hardika Catur S <ha...@solusi247.com.INVALID> on 2016/06/22 08:54:30 UTC
Automatic Language Identification
Hi,
I will make the collection in the collection solrcloud and used for
"automatic language identification" but when they failed to make a
collection in his process:
1. The automatic language identification
ERROR: Failed to create collection 'coba' due to:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:Error
from server at http://localhost:8983/solr: Error CREATEing SolrCore
'coba_shard1_replica1': Unable to create core [coba_shard1_replica1]
Caused by:
org.apache.solr.update.processor.TikaLanguageIdentifierUpdateProcessorFactory
The following script in solrconfig.xml :
<updateRequestProcessorChain name="langid">
<processor name="langid"
class="org.apache.solr.update.processor.TikaLanguageIdentifierUpdateProcessorFactory">
<lst name="defaults">
<str name="langid.fl">title, content</str>
<str name="langid.langField">language</str>
<str name="langid.langsField">languages</str>
</lst>
</processor>
</updateRequestProcessorChain>
<requestHandler name="/update" class="solr.UpdateRequestHandler">
<lst name="invariants">
<str name="update.chain">langid</str>
</lst>
</requestHandler>
2. Multi Language
ERROR: Failed to create collection 'coba' due to:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:Error
from server at http://localhost:8983/solr: Error CREATEing SolrCore
'coba_shard1_replica1': Unable to create core [coba_shard1_replica1]
Caused by: sia.ch14.MultiTextField
The following script in schema.xml :
<fieldType name="text"
class="sia.ch14.MultiTextField" sortMissingLast="true"
defaultFieldType="text_id"
fieldMappings="en:text_en"/>
please help me,
Thanks,
Hardika CS.
Re: Automatic Language Identification
Posted by Alexandre Rafalovitch <ar...@gmail.com>.
In both cases, the issues seems to be related to the library not being
loaded. For Tika identifier, I believe it is
solr-langid-<version>.jar, for the sia.* it is whatever the book
recommended.
Are you running SolrCloud? Additional libraries are slightly
complicated with that, you need to make sure they are present on all
the servers this is deployed to.
Regards,
Alex.
----
Newsletter and resources for Solr beginners and intermediates:
http://www.solr-start.com/
On 22 June 2016 at 18:54, Hardika Catur S
<ha...@solusi247.com.invalid> wrote:
> Hi,
>
> I will make the collection in the collection solrcloud and used for
> "automatic language identification" but when they failed to make a
> collection in his process:
>
> 1. The automatic language identification
>
>
> ERROR: Failed to create collection 'coba' due to:
> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:Error
> from server at http://localhost:8983/solr: Error CREATEing SolrCore
> 'coba_shard1_replica1': Unable to create core [coba_shard1_replica1] Caused
> by:
> org.apache.solr.update.processor.TikaLanguageIdentifierUpdateProcessorFactory
>
> The following script in solrconfig.xml :
>
> <updateRequestProcessorChain name="langid">
> <processor name="langid"
> class="org.apache.solr.update.processor.TikaLanguageIdentifierUpdateProcessorFactory">
> <lst name="defaults">
> <str name="langid.fl">title, content</str>
> <str name="langid.langField">language</str>
> <str name="langid.langsField">languages</str>
> </lst>
> </processor>
> </updateRequestProcessorChain>
>
> <requestHandler name="/update" class="solr.UpdateRequestHandler">
> <lst name="invariants">
> <str name="update.chain">langid</str>
> </lst>
> </requestHandler>
>
>
>
> 2. Multi Language
>
> ERROR: Failed to create collection 'coba' due to:
> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:Error
> from server at http://localhost:8983/solr: Error CREATEing SolrCore
> 'coba_shard1_replica1': Unable to create core [coba_shard1_replica1] Caused
> by: sia.ch14.MultiTextField
>
> The following script in schema.xml :
>
> <fieldType name="text"
> class="sia.ch14.MultiTextField" sortMissingLast="true"
> defaultFieldType="text_id"
> fieldMappings="en:text_en"/>
>
> please help me,
> Thanks,
> Hardika CS.
>
Re: Automatic Language Identification
Posted by William Bell <bi...@gmail.com>.
We should add a simple filter in Solr for this. The current way requires
indexing.
https://github.com/kkrugler/yalder is good, it would be a great filter:
if NOT english, fail the whole text.
On Fri, Jul 1, 2016 at 6:33 AM, Allison, Timothy B. <ta...@mitre.org>
wrote:
> +1 to langdetect
>
> In Tika 2.0, we're going to remove our own language detection code and
> allow users to select Optimaize (fork of langdetect), MIT Lincoln Lab’s
> Text.jl library or Yalder (https://github.com/kkrugler/yalder). The
> first two are now available in Tika 1.13.
>
> -----Original Message-----
> From: Markus Jelsma [mailto:markus.jelsma@openindex.io]
> Sent: Wednesday, June 22, 2016 8:27 AM
> To: solr-user@lucene.apache.org; solr-user <so...@lucene.apache.org>
> Subject: RE: Automatic Language Identification
>
> Hello,
>
> I recommend using the langdetect language detector, it supports many more
> languages and has much higher precission than Tika's detector.
>
> Markus
>
>
>
--
Bill Bell
billnbell@gmail.com
cell 720-256-8076
RE: Automatic Language Identification
Posted by "Allison, Timothy B." <ta...@mitre.org>.
+1 to langdetect
In Tika 2.0, we're going to remove our own language detection code and allow users to select Optimaize (fork of langdetect), MIT Lincoln Lab’s Text.jl library or Yalder (https://github.com/kkrugler/yalder). The first two are now available in Tika 1.13.
-----Original Message-----
From: Markus Jelsma [mailto:markus.jelsma@openindex.io]
Sent: Wednesday, June 22, 2016 8:27 AM
To: solr-user@lucene.apache.org; solr-user <so...@lucene.apache.org>
Subject: RE: Automatic Language Identification
Hello,
I recommend using the langdetect language detector, it supports many more languages and has much higher precission than Tika's detector.
Markus
RE: Automatic Language Identification
Posted by Markus Jelsma <ma...@openindex.io>.
Hello,
I recommend using the langdetect language detector, it supports many more languages and has much higher precission than Tika's detector.
Markus
-----Original message-----
> From:Alexandre Rafalovitch <ar...@gmail.com>
> Sent: Wednesday 22nd June 2016 12:32
> To: solr-user <so...@lucene.apache.org>
> Subject: Re: Automatic Language Identification
>
> In both cases, the issues seems to be related to the library not being
> loaded. For Tika identifier, I believe it is
> solr-langid-<version>.jar, for the sia.* it is whatever the book
> recommended.
>
> Are you running SolrCloud? Additional libraries are slightly
> complicated with that, you need to make sure they are present on all
> the servers this is deployed to.
>
> Regards,
> Alex.
> ----
> Newsletter and resources for Solr beginners and intermediates:
> http://www.solr-start.com/
>
>
> On 22 June 2016 at 18:54, Hardika Catur S
> <ha...@solusi247.com.invalid> wrote:
> > Hi,
> >
> > I will make the collection in the collection solrcloud and used for
> > "automatic language identification" but when they failed to make a
> > collection in his process:
> >
> > 1. The automatic language identification
> >
> >
> > ERROR: Failed to create collection 'coba' due to:
> > org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:Error
> > from server at http://localhost:8983/solr: Error CREATEing SolrCore
> > 'coba_shard1_replica1': Unable to create core [coba_shard1_replica1] Caused
> > by:
> > org.apache.solr.update.processor.TikaLanguageIdentifierUpdateProcessorFactory
> >
> > The following script in solrconfig.xml :
> >
> > <updateRequestProcessorChain name="langid">
> > <processor name="langid"
> > class="org.apache.solr.update.processor.TikaLanguageIdentifierUpdateProcessorFactory">
> > <lst name="defaults">
> > <str name="langid.fl">title, content</str>
> > <str name="langid.langField">language</str>
> > <str name="langid.langsField">languages</str>
> > </lst>
> > </processor>
> > </updateRequestProcessorChain>
> >
> > <requestHandler name="/update" class="solr.UpdateRequestHandler">
> > <lst name="invariants">
> > <str name="update.chain">langid</str>
> > </lst>
> > </requestHandler>
> >
> >
> >
> > 2. Multi Language
> >
> > ERROR: Failed to create collection 'coba' due to:
> > org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:Error
> > from server at http://localhost:8983/solr: Error CREATEing SolrCore
> > 'coba_shard1_replica1': Unable to create core [coba_shard1_replica1] Caused
> > by: sia.ch14.MultiTextField
> >
> > The following script in schema.xml :
> >
> > <fieldType name="text"
> > class="sia.ch14.MultiTextField" sortMissingLast="true"
> > defaultFieldType="text_id"
> > fieldMappings="en:text_en"/>
> >
> > please help me,
> > Thanks,
> > Hardika CS.
> >
>