You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by adfel70 <ad...@gmail.com> on 2013/07/07 18:05:41 UTC
lang.fallback doesn't work when using lang.fallbackFields
Hi
I'm trying to index a set of documents with solr's language detection
component.
I set
<langid.fallbackFields>user_lan</langid.fallbackFields>
<langid.whitelist>en,it</langid.whitelist>
<langid.fallback>en</langid.fallback>
In some documents user_lan has 'sk', solr falls-back to 'sk' ,which is not
in the whitelist, and instead of falling back to 'en' as stated here
<http://wiki.apache.org/solr/LanguageDetection#langid.fallbackFields> , I
get an excpetion regarding not having a text_sk field in the schema.
Anyone encountered this behavior?
thanks.
--
View this message in context: http://lucene.472066.n3.nabble.com/lang-fallback-doesn-t-work-when-using-lang-fallbackFields-tp4076048.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: lang.fallback doesn't work when using lang.fallbackFields
Posted by Sam Dillingham <sa...@gmail.com>.
unsubscribe
On Sun, Jul 28, 2013 at 5:59 PM, Jan Høydahl <ja...@cominvent.com> wrote:
> Hi,
>
> Looking at the code, you are right. Whitelist processing is only done on
> detected languages, not on the fallback or fallbackFields languages, since
> these are assumed to be correct. Thus you should not pass in a fallback
> language, either in the input document or with langid.fallback which cannot
> be handled by your schema.
>
> This is by design. However, I can also see an argument for making
> fallbackFields subject to whitelist logic, especially if you do not control
> the application that populates this field, to safeguard against exception.
> Also, such a change woudl not harm any of the existing functionality, so it
> would be safe to introduce.
>
> Feel free to write a JIRA issue for it.
>
> A workaround could be to write a simple UpdateProcessor which removes any
> illegal value from langid.fallbackFields before the LangId processor.
>
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.cominvent.com
>
> 7. juli 2013 kl. 18:05 skrev adfel70 <ad...@gmail.com>:
>
> > Hi
> > I'm trying to index a set of documents with solr's language detection
> > component.
> > I set
> > <langid.fallbackFields>user_lan</langid.fallbackFields>
> > <langid.whitelist>en,it</langid.whitelist>
> > <langid.fallback>en</langid.fallback>
> >
> > In some documents user_lan has 'sk', solr falls-back to 'sk' ,which is
> not
> > in the whitelist, and instead of falling back to 'en' as stated here
> > <http://wiki.apache.org/solr/LanguageDetection#langid.fallbackFields>
> , I
> > get an excpetion regarding not having a text_sk field in the schema.
> >
> > Anyone encountered this behavior?
> >
> > thanks.
> >
> >
> >
> > --
> > View this message in context:
> http://lucene.472066.n3.nabble.com/lang-fallback-doesn-t-work-when-using-lang-fallbackFields-tp4076048.html
> > Sent from the Solr - User mailing list archive at Nabble.com.
>
>
--
Sam
Re: lang.fallback doesn't work when using lang.fallbackFields
Posted by Jan Høydahl <ja...@cominvent.com>.
Hi,
Looking at the code, you are right. Whitelist processing is only done on detected languages, not on the fallback or fallbackFields languages, since these are assumed to be correct. Thus you should not pass in a fallback language, either in the input document or with langid.fallback which cannot be handled by your schema.
This is by design. However, I can also see an argument for making fallbackFields subject to whitelist logic, especially if you do not control the application that populates this field, to safeguard against exception. Also, such a change woudl not harm any of the existing functionality, so it would be safe to introduce.
Feel free to write a JIRA issue for it.
A workaround could be to write a simple UpdateProcessor which removes any illegal value from langid.fallbackFields before the LangId processor.
--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com
7. juli 2013 kl. 18:05 skrev adfel70 <ad...@gmail.com>:
> Hi
> I'm trying to index a set of documents with solr's language detection
> component.
> I set
> <langid.fallbackFields>user_lan</langid.fallbackFields>
> <langid.whitelist>en,it</langid.whitelist>
> <langid.fallback>en</langid.fallback>
>
> In some documents user_lan has 'sk', solr falls-back to 'sk' ,which is not
> in the whitelist, and instead of falling back to 'en' as stated here
> <http://wiki.apache.org/solr/LanguageDetection#langid.fallbackFields> , I
> get an excpetion regarding not having a text_sk field in the schema.
>
> Anyone encountered this behavior?
>
> thanks.
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/lang-fallback-doesn-t-work-when-using-lang-fallbackFields-tp4076048.html
> Sent from the Solr - User mailing list archive at Nabble.com.