You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by HSingh <hs...@gmail.com> on 2010/07/16 04:41:40 UTC
Novice seeking help to change filters to search without diacritics
I am new to Solr and seeking your help to change filter from
ISOLatin1AccentFilterFactory to ASCIIFoldingFilterFactory files. I am not
sure what change is to be made and where exactly this change is to be made.
And finally, what would replace mapping-ISOLatin1Accent.txt file? I would
like Solr to search both with and without diacritics found in
transliteration of Indian languages with characters such as Ā ś ṛ ṇ, etc.
The three files that are currently being used are attached. I would deeply
appreciate your help. Thank you.
http://lucene.472066.n3.nabble.com/file/n971263/schema.xml schema.xml
http://lucene.472066.n3.nabble.com/file/n971263/solrconfig.xml
solrconfig.xml
http://lucene.472066.n3.nabble.com/file/n971263/mapping-ISOLatin1Accent.txt
mapping-ISOLatin1Accent.txt
--
View this message in context: http://lucene.472066.n3.nabble.com/Novice-seeking-help-to-change-filters-to-search-without-diacritics-tp971263p971263.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Novice seeking help to change filters to search without
diacritics
Posted by Erick Erickson <er...@gmail.com>.
use copyfield in your schema file. The copyfield takes its own
analyzer, so the original can fold and the copy may not.
dismax might help you at query time on this...
HTH
Erick
On Sat, Jul 24, 2010 at 11:40 PM, HSingh <hs...@gmail.com> wrote:
>
>
> : Usually people set up two fields, one with diacritics and one without.
> : Then searches are against both fields. If you think a match against the
> field
> : with diacritics is more valuable, you can give that field a boost.
>
> Hi Steve, where can one setup these two fields? Thank you for your kind
> assistance!
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Novice-seeking-help-to-change-filters-to-search-without-diacritics-tp971263p993150.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
RE: Novice seeking help to change filters to search without
diacritics
Posted by HSingh <hs...@gmail.com>.
: Usually people set up two fields, one with diacritics and one without.
: Then searches are against both fields. If you think a match against the
field
: with diacritics is more valuable, you can give that field a boost.
Hi Steve, where can one setup these two fields? Thank you for your kind
assistance!
--
View this message in context: http://lucene.472066.n3.nabble.com/Novice-seeking-help-to-change-filters-to-search-without-diacritics-tp971263p993150.html
Sent from the Solr - User mailing list archive at Nabble.com.
RE: Novice seeking help to change filters to search without
diacritics
Posted by Steven A Rowe <sa...@syr.edu>.
Hi HSingh,
Usually people set up two fields, one with diacritics and one without. Then searches are against both fields. If you think a match against the field with diacritics is more valuable, you can give that field a boost.
Steve
> -----Original Message-----
> From: HSingh [mailto:hsingh1@gmail.com]
> Sent: Friday, July 23, 2010 5:20 PM
> To: solr-user@lucene.apache.org
> Subject: RE: Novice seeking help to change filters to search without
> diacritics
>
>
> Hi Steve, This is extremely helpful! What is the best way to also
> preserve/append the diacritics in the index in case someone searches using
> them? I deeply appreciate your help!
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Novice-
> seeking-help-to-change-filters-to-search-without-diacritics-
> tp971263p990949.html
> Sent from the Solr - User mailing list archive at Nabble.com.
RE: Novice seeking help to change filters to search without
diacritics
Posted by HSingh <hs...@gmail.com>.
Hi Steve, This is extremely helpful! What is the best way to also
preserve/append the diacritics in the index in case someone searches using
them? I deeply appreciate your help!
--
View this message in context: http://lucene.472066.n3.nabble.com/Novice-seeking-help-to-change-filters-to-search-without-diacritics-tp971263p990949.html
Sent from the Solr - User mailing list archive at Nabble.com.
RE: Novice seeking help to change filters to search without
diacritics
Posted by Steven A Rowe <sa...@syr.edu>.
Hi HSingh,
Maybe the mapping file I attached to https://issues.apache.org/jira/browse/SOLR-2013 will help?
Steve
> -----Original Message-----
> From: HSingh [mailto:hsingh1@gmail.com]
> Sent: Thursday, July 22, 2010 11:30 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Novice seeking help to change filters to search without
> diacritics
>
>
> Hoss, thank you for your helpful response!
>
> : i think what's confusing you is that you are using the
> : MappingCharFilterFactory with that file in your "text" field type to
> : convert any ISOLatin1Accent characters to their "base" characters
>
> The problem is that a large range of characters are not getting converting
> to their base characters. The ASCIIFoldingFilterFactory handles this
> conversion for the entire Latin character set, including the extended sets
> without having to specify individual characters and their equivalent base
> characters.
>
> Is there way for me to switch to ASCIIFoldingFilterFactory? If so, what
> changes do I need to make to these files? I would appreciate your help!
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Novice-
> seeking-help-to-change-filters-to-search-without-diacritics-
> tp971263p988890.html
> Sent from the Solr - User mailing list archive at Nabble.com.
Re: Novice seeking help to change filters to search without
diacritics
Posted by HSingh <hs...@gmail.com>.
Hoss, thank you for your helpful response!
: i think what's confusing you is that you are using the
: MappingCharFilterFactory with that file in your "text" field type to
: convert any ISOLatin1Accent characters to their "base" characters
The problem is that a large range of characters are not getting converting
to their base characters. The ASCIIFoldingFilterFactory handles this
conversion for the entire Latin character set, including the extended sets
without having to specify individual characters and their equivalent base
characters.
Is there way for me to switch to ASCIIFoldingFilterFactory? If so, what
changes do I need to make to these files? I would appreciate your help!
--
View this message in context: http://lucene.472066.n3.nabble.com/Novice-seeking-help-to-change-filters-to-search-without-diacritics-tp971263p988890.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Novice seeking help to change filters to search without
diacritics
Posted by Chris Hostetter <ho...@fucit.org>.
: I am new to Solr and seeking your help to change filter from
: ISOLatin1AccentFilterFactory to ASCIIFoldingFilterFactory files. I am not
According to the files you posted, you aren't using the
ISOLatin1AccentFilterFactory -- so problem solved w/o making any changes.
: sure what change is to be made and where exactly this change is to be made.
: And finally, what would replace mapping-ISOLatin1Accent.txt file? I would
i think what's confusing you is thta you are using the
MappingCharFilterFactory with that file in your "text" field type to
convert any ISOLatin1Accent characters to their "base" characters (i'm
sure there is a more precise term for it, but i'm not a charset savant
like rmuir so i odn't know what it's caled)
: like Solr to search both with and without diacritics found in
: transliteration of Indian languages with characters such as Ā ś ṛ ṇ, etc.
your existing usage should allow that on any fields using the "text" type
-- if you index those characters they will get "flattened" and if someone
searches on those characters they will get "flattened" -- it's just like
using LowerCaseFilter -- as long as you do it at index and query time
everything is consistent.
if you want docs to score higher when even the accents match, just index
and query across two fields: on with that charfilter and one w/o.
-Hoss