You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Søren <sd...@syntonetic.com> on 2015/07/02 10:17:59 UTC

accent insensitive field-type

Hi Solr users

I'm new to Solr and I need to be able to search in structured data in a 
case and accent insensitive manner. E.g. find "Crème brûlée", both when 
quering with "Crème brûlée" and "creme brulee".

It seems that none of the build-in text types support this, or am I wrong?
So I try to add my own inspired by another post, although it was old.

I'm running solr-5.2.1.

Curl to http://localhost:8983/solr/mycore/schema
{
"add-field-type":{
      "name":"myTxtField",
      "class":"solr.TextField",
      "positionIncrementGap":"100",
      "analyzer":{
         "charFilter": {"class":"solr.MappingCharFilterFactory", 
"mapping":"mapping-ISOLatin1Accent.txt"},
         "filter": {"class":"solr.LowerCaseFilterFactory"},
         "tokenizer": {"class":"solr.StandardTokenizerFactory"}
         }
     }
}

But it doesn't work and when I look in '[... 
]\solr-5.2.1\server\solr\mycore\conf\managed-schema'
the analyzer section is reduced to this:
   <fieldType name="myTxtField" class="solr.TextField" 
positionIncrementGap="100">
     <analyzer>
       <tokenizer class="solr.StandardTokenizerFactory"/>
     </analyzer>
   </fieldType>

  I'm I almost there or am I on a completely wrong track?

Thanks in advance
Søren


Re: accent insensitive field-type

Posted by Erick Erickson <er...@gmail.com>.
bq: q=brulee        didn't find anything. It goes into to the raw index I guess

The first thing to do in this case is add &debug=query to the URL and look
at the parsed query near the end of the return packet. In this case it would
look something like

q=default_search_field:brulee

where deault_search_field is the "df" specification in the request
handler in solrconfig.xml.

Best,
Erick

On Fri, Jul 3, 2015 at 1:27 AM, Søren <sd...@syntonetic.com> wrote:
> Thanks Steve! Everything works now.
> A little modification:
>
>         "analyzer":{
>             "charFilters": [ {"class":"solr.MappingCharFilterFactory",
> "mapping":"mapping-ISOLatin1Accent.txt"} ],
>             "tokenizer": {"class":"solr.StandardTokenizerFactory"},
>             "filters": [{"class":"solr.LowerCaseFilterFactory"}]
>         }
>
> Thankfully, when key is a plural word, the value is an array.
>
> It was still teasing me when I tested with various queries. But specifying
> field solved that for me too.
>
> ...q=brulee        didn't find anything. It goes into to the raw index I
> guess
>
> ...q=desert:brulee        did find "Crème brûlée"!
>
> cheers
> Søren
>
>
>
> On 02-07-2015 17:31, Steve Rowe wrote:
>>
>> Hi Søren,
>>
>> “charFilter” should be “charFilters”, and “filter” should be “filters”;
>> and both their values should be arrays - try this:
>>
>> {
>>    "add-field-type”: {
>>      "name":"myTxtField",
>>      "class":"solr.TextField",
>>      "positionIncrementGap":"100",
>>      "analyzer”: {
>>        "charFilters": [ {"class":"solr.MappingCharFilterFactory",
>> "mapping":"mapping-ISOLatin1Accent.txt”} ],
>>        "tokenizer": [ {"class":"solr.StandardTokenizerFactory”} ],
>>        "filters": {"class":"solr.LowerCaseFilterFactory"}
>>      }
>>    }
>> }
>>
>> There should be better error messages for misspellings here.  I’ll file a
>> JIRA issue.
>>
>> (I also moved “filters” after “tokenizer” since that’s the order in which
>> they’re executed in an analysis pipeline, but Solr will interpret the
>> out-of-order version correctly.)
>>
>> FYI, if you want to *correct* a field type, rather than create a new one,
>> you should use the “replace-field-type” command instead of the
>> “add-field-type” command.  You’ll get an error if you attempt to add a field
>> type that already exists in the schema.
>>
>> Steve
>>
>>> On Jul 2, 2015, at 1:17 AM, Søren <sd...@syntonetic.com> wrote:
>>>
>>> Hi Solr users
>>>
>>> I'm new to Solr and I need to be able to search in structured data in a
>>> case and accent insensitive manner. E.g. find "Crème brûlée", both when
>>> quering with "Crème brûlée" and "creme brulee".
>>>
>>> It seems that none of the build-in text types support this, or am I
>>> wrong?
>>> So I try to add my own inspired by another post, although it was old.
>>>
>>> I'm running solr-5.2.1.
>>>
>>> Curl to http://localhost:8983/solr/mycore/schema
>>> {
>>> "add-field-type":{
>>>      "name":"myTxtField",
>>>      "class":"solr.TextField",
>>>      "positionIncrementGap":"100",
>>>      "analyzer":{
>>>         "charFilter": {"class":"solr.MappingCharFilterFactory",
>>> "mapping":"mapping-ISOLatin1Accent.txt"},
>>>         "filter": {"class":"solr.LowerCaseFilterFactory"},
>>>         "tokenizer": {"class":"solr.StandardTokenizerFactory"}
>>>         }
>>>     }
>>> }
>>>
>>> But it doesn't work and when I look in '[...
>>> ]\solr-5.2.1\server\solr\mycore\conf\managed-schema'
>>> the analyzer section is reduced to this:
>>>   <fieldType name="myTxtField" class="solr.TextField"
>>> positionIncrementGap="100">
>>>     <analyzer>
>>>       <tokenizer class="solr.StandardTokenizerFactory"/>
>>>     </analyzer>
>>>   </fieldType>
>>>
>>> I'm I almost there or am I on a completely wrong track?
>>>
>>> Thanks in advance
>>> Søren
>>>
>

Re: accent insensitive field-type

Posted by Steve Rowe <sa...@gmail.com>.
Hi Søren,

> On Jul 3, 2015, at 4:27 AM, Søren <sd...@syntonetic.com> wrote:
> 
> Thanks Steve! Everything works now.
> A little modification:
> 
>        "analyzer":{
>            "charFilters": [ {"class":"solr.MappingCharFilterFactory", "mapping":"mapping-ISOLatin1Accent.txt"} ],
>            "tokenizer": {"class":"solr.StandardTokenizerFactory"},
>            "filters": [{"class":"solr.LowerCaseFilterFactory"}]
>        }

I’m glad you got it to work.

Yeah, I put square brackets in the wrong place, cool you figured it out and fixed it.

> Thankfully, when key is a plural word, the value is an array.
> 
> It was still teasing me when I tested with various queries. But specifying field solved that for me too.
> 
> ...q=brulee        didn't find anything. It goes into to the raw index I guess
> 
> ...q=desert:brulee        did find "Crème brûlée”!

In your query request handler you should specify a “df” param (default field), likely under the defaults section (so that it can be overridden via per-request param) - this param will work with the dismax, edisma, or wtandard query parsers.  The “qf" param, which supports a list of query fields (and field aliasing)[1], also works in the dismax and edismax query parsers.

Steve

[1] See section "Field aliasing using per-field qf overrides” on the edismax ref guide page: <https://cwiki.apache.org/confluence/display/solr/The+Extended+DisMax+Query+Parser> and the qf param  description on the dismax ref guide page: <https://cwiki.apache.org/confluence/display/solr/The+DisMax+Query+Parser#TheDisMaxQueryParser-DisMaxParameters>.

Re: accent insensitive field-type

Posted by Søren <sd...@syntonetic.com>.
Thanks Steve! Everything works now.
A little modification:

         "analyzer":{
             "charFilters": [ {"class":"solr.MappingCharFilterFactory", 
"mapping":"mapping-ISOLatin1Accent.txt"} ],
             "tokenizer": {"class":"solr.StandardTokenizerFactory"},
             "filters": [{"class":"solr.LowerCaseFilterFactory"}]
         }

Thankfully, when key is a plural word, the value is an array.

It was still teasing me when I tested with various queries. But 
specifying field solved that for me too.

...q=brulee        didn't find anything. It goes into to the raw index I 
guess

...q=desert:brulee        did find "Crème brûlée"!

cheers
Søren


On 02-07-2015 17:31, Steve Rowe wrote:
> Hi Søren,
>
> “charFilter” should be “charFilters”, and “filter” should be “filters”; and both their values should be arrays - try this:
>
> {
>    "add-field-type”: {
>      "name":"myTxtField",
>      "class":"solr.TextField",
>      "positionIncrementGap":"100",
>      "analyzer”: {
>        "charFilters": [ {"class":"solr.MappingCharFilterFactory", "mapping":"mapping-ISOLatin1Accent.txt”} ],
>        "tokenizer": [ {"class":"solr.StandardTokenizerFactory”} ],
>        "filters": {"class":"solr.LowerCaseFilterFactory"}
>      }
>    }
> }
>
> There should be better error messages for misspellings here.  I’ll file a JIRA issue.
>
> (I also moved “filters” after “tokenizer” since that’s the order in which they’re executed in an analysis pipeline, but Solr will interpret the out-of-order version correctly.)
>
> FYI, if you want to *correct* a field type, rather than create a new one, you should use the “replace-field-type” command instead of the “add-field-type” command.  You’ll get an error if you attempt to add a field type that already exists in the schema.
>
> Steve
>
>> On Jul 2, 2015, at 1:17 AM, Søren <sd...@syntonetic.com> wrote:
>>
>> Hi Solr users
>>
>> I'm new to Solr and I need to be able to search in structured data in a case and accent insensitive manner. E.g. find "Crème brûlée", both when quering with "Crème brûlée" and "creme brulee".
>>
>> It seems that none of the build-in text types support this, or am I wrong?
>> So I try to add my own inspired by another post, although it was old.
>>
>> I'm running solr-5.2.1.
>>
>> Curl to http://localhost:8983/solr/mycore/schema
>> {
>> "add-field-type":{
>>      "name":"myTxtField",
>>      "class":"solr.TextField",
>>      "positionIncrementGap":"100",
>>      "analyzer":{
>>         "charFilter": {"class":"solr.MappingCharFilterFactory", "mapping":"mapping-ISOLatin1Accent.txt"},
>>         "filter": {"class":"solr.LowerCaseFilterFactory"},
>>         "tokenizer": {"class":"solr.StandardTokenizerFactory"}
>>         }
>>     }
>> }
>>
>> But it doesn't work and when I look in '[... ]\solr-5.2.1\server\solr\mycore\conf\managed-schema'
>> the analyzer section is reduced to this:
>>   <fieldType name="myTxtField" class="solr.TextField" positionIncrementGap="100">
>>     <analyzer>
>>       <tokenizer class="solr.StandardTokenizerFactory"/>
>>     </analyzer>
>>   </fieldType>
>>
>> I'm I almost there or am I on a completely wrong track?
>>
>> Thanks in advance
>> Søren
>>


Re: accent insensitive field-type

Posted by Steve Rowe <sa...@gmail.com>.
See https://issues.apache.org/jira/browse/SOLR-7749

> On Jul 2, 2015, at 8:31 AM, Steve Rowe <sa...@gmail.com> wrote:
> 
> Hi Søren,
> 
> “charFilter” should be “charFilters”, and “filter” should be “filters”; and both their values should be arrays - try this:
> 
> {
>  "add-field-type”: {
>    "name":"myTxtField",
>    "class":"solr.TextField",
>    "positionIncrementGap":"100",
>    "analyzer”: {
>      "charFilters": [ {"class":"solr.MappingCharFilterFactory", "mapping":"mapping-ISOLatin1Accent.txt”} ],
>      "tokenizer": [ {"class":"solr.StandardTokenizerFactory”} ],
>      "filters": {"class":"solr.LowerCaseFilterFactory"}
>    }
>  }
> }
> 
> There should be better error messages for misspellings here.  I’ll file a JIRA issue.
> 
> (I also moved “filters” after “tokenizer” since that’s the order in which they’re executed in an analysis pipeline, but Solr will interpret the out-of-order version correctly.)
> 
> FYI, if you want to *correct* a field type, rather than create a new one, you should use the “replace-field-type” command instead of the “add-field-type” command.  You’ll get an error if you attempt to add a field type that already exists in the schema.
> 
> Steve
> 
>> On Jul 2, 2015, at 1:17 AM, Søren <sd...@syntonetic.com> wrote:
>> 
>> Hi Solr users
>> 
>> I'm new to Solr and I need to be able to search in structured data in a case and accent insensitive manner. E.g. find "Crème brûlée", both when quering with "Crème brûlée" and "creme brulee".
>> 
>> It seems that none of the build-in text types support this, or am I wrong?
>> So I try to add my own inspired by another post, although it was old.
>> 
>> I'm running solr-5.2.1.
>> 
>> Curl to http://localhost:8983/solr/mycore/schema
>> {
>> "add-field-type":{
>>    "name":"myTxtField",
>>    "class":"solr.TextField",
>>    "positionIncrementGap":"100",
>>    "analyzer":{
>>       "charFilter": {"class":"solr.MappingCharFilterFactory", "mapping":"mapping-ISOLatin1Accent.txt"},
>>       "filter": {"class":"solr.LowerCaseFilterFactory"},
>>       "tokenizer": {"class":"solr.StandardTokenizerFactory"}
>>       }
>>   }
>> }
>> 
>> But it doesn't work and when I look in '[... ]\solr-5.2.1\server\solr\mycore\conf\managed-schema'
>> the analyzer section is reduced to this:
>> <fieldType name="myTxtField" class="solr.TextField" positionIncrementGap="100">
>>   <analyzer>
>>     <tokenizer class="solr.StandardTokenizerFactory"/>
>>   </analyzer>
>> </fieldType>
>> 
>> I'm I almost there or am I on a completely wrong track?
>> 
>> Thanks in advance
>> Søren
>> 
> 


Re: accent insensitive field-type

Posted by Steve Rowe <sa...@gmail.com>.
Hi Søren,

“charFilter” should be “charFilters”, and “filter” should be “filters”; and both their values should be arrays - try this:

{
  "add-field-type”: {
    "name":"myTxtField",
    "class":"solr.TextField",
    "positionIncrementGap":"100",
    "analyzer”: {
      "charFilters": [ {"class":"solr.MappingCharFilterFactory", "mapping":"mapping-ISOLatin1Accent.txt”} ],
      "tokenizer": [ {"class":"solr.StandardTokenizerFactory”} ],
      "filters": {"class":"solr.LowerCaseFilterFactory"}
    }
  }
}

There should be better error messages for misspellings here.  I’ll file a JIRA issue.

(I also moved “filters” after “tokenizer” since that’s the order in which they’re executed in an analysis pipeline, but Solr will interpret the out-of-order version correctly.)

FYI, if you want to *correct* a field type, rather than create a new one, you should use the “replace-field-type” command instead of the “add-field-type” command.  You’ll get an error if you attempt to add a field type that already exists in the schema.

Steve

> On Jul 2, 2015, at 1:17 AM, Søren <sd...@syntonetic.com> wrote:
> 
> Hi Solr users
> 
> I'm new to Solr and I need to be able to search in structured data in a case and accent insensitive manner. E.g. find "Crème brûlée", both when quering with "Crème brûlée" and "creme brulee".
> 
> It seems that none of the build-in text types support this, or am I wrong?
> So I try to add my own inspired by another post, although it was old.
> 
> I'm running solr-5.2.1.
> 
> Curl to http://localhost:8983/solr/mycore/schema
> {
> "add-field-type":{
>     "name":"myTxtField",
>     "class":"solr.TextField",
>     "positionIncrementGap":"100",
>     "analyzer":{
>        "charFilter": {"class":"solr.MappingCharFilterFactory", "mapping":"mapping-ISOLatin1Accent.txt"},
>        "filter": {"class":"solr.LowerCaseFilterFactory"},
>        "tokenizer": {"class":"solr.StandardTokenizerFactory"}
>        }
>    }
> }
> 
> But it doesn't work and when I look in '[... ]\solr-5.2.1\server\solr\mycore\conf\managed-schema'
> the analyzer section is reduced to this:
>  <fieldType name="myTxtField" class="solr.TextField" positionIncrementGap="100">
>    <analyzer>
>      <tokenizer class="solr.StandardTokenizerFactory"/>
>    </analyzer>
>  </fieldType>
> 
> I'm I almost there or am I on a completely wrong track?
> 
> Thanks in advance
> Søren
> 


Re: accent insensitive field-type

Posted by Shawn Heisey <ap...@elyograg.org>.
On 7/2/2015 8:53 AM, Søren wrote:
> I'm trying the add the ICUFoldingFilterFactory in the analyzer.
> I suspect that my problem is that the filter class doesn't load.
> The managed-schema file is the same info as when looking in the schema
> browser in the web gui.

The ICU analysis components are considered "contrib" and therefore
require adding jars to Solr for them to work.  The notes on the
AnalyzersTokenizersTokenFilters wiki page tell you where to look for
more information about those jars.

http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters#solr.ICUFoldingFilterFactory

The best place to add additional jars is ${solr.solr.home}/lib ... a
directory which usually must be created.  The solr home is where
solr.xml lives, providing global config options to Solr.  The default
location for the solr home in the 5.x download is server/solr, but that
can be overridden.

Thanks,
Shawn


Re: accent insensitive field-type

Posted by Søren <sd...@syntonetic.com>.
Thanks Ahmet

I'm trying the add the ICUFoldingFilterFactory in the analyzer.
I suspect that my problem is that the filter class doesn't load.
The managed-schema file is the same info as when looking in the schema 
browser in the web gui.

Cheers

On 02-07-2015 10:47, Ahmet Arslan wrote:
> Hi Soren,
>
> I am not familiar with managed schema part, but there are built-in filters for this task.
>
> ASCIIFoldingFilter and ICUFoldingFilter are two examples.
>
> Also solr provides two files: mapping-FoldToASCII.txt and mapping-ISOLatin1Accent.txt to be used with
> MappingCharFilter as you did.
> You are probably hitting a problem with managed schema.
>
> Ahmet
>
>
> On Thursday, July 2, 2015 11:17 AM, Søren <sd...@syntonetic.com> wrote:
> Hi Solr users
>
> I'm new to Solr and I need to be able to search in structured data in a
> case and accent insensitive manner. E.g. find "Crème brûlée", both when
> quering with "Crème brûlée" and "creme brulee".
>
> It seems that none of the build-in text types support this, or am I wrong?
> So I try to add my own inspired by another post, although it was old.
>
> I'm running solr-5.2.1.
>
> Curl to http://localhost:8983/solr/mycore/schema
> {
> "add-field-type":{
>        "name":"myTxtField",
>        "class":"solr.TextField",
>        "positionIncrementGap":"100",
>        "analyzer":{
>           "charFilter": {"class":"solr.MappingCharFilterFactory",
> "mapping":"mapping-ISOLatin1Accent.txt"},
>           "filter": {"class":"solr.LowerCaseFilterFactory"},
>           "tokenizer": {"class":"solr.StandardTokenizerFactory"}
>           }
>       }
> }
>
> But it doesn't work and when I look in '[...
> ]\solr-5.2.1\server\solr\mycore\conf\managed-schema'
> the analyzer section is reduced to this:
>     <fieldType name="myTxtField" class="solr.TextField"
> positionIncrementGap="100">
>       <analyzer>
>         <tokenizer class="solr.StandardTokenizerFactory"/>
>       </analyzer>
>     </fieldType>
>
>    I'm I almost there or am I on a completely wrong track?
>
> Thanks in advance
> Søren


Re: accent insensitive field-type

Posted by Ahmet Arslan <io...@yahoo.com.INVALID>.
Hi Soren,

I am not familiar with managed schema part, but there are built-in filters for this task.

ASCIIFoldingFilter and ICUFoldingFilter are two examples. 

Also solr provides two files: mapping-FoldToASCII.txt and mapping-ISOLatin1Accent.txt to be used with 
MappingCharFilter as you did.
You are probably hitting a problem with managed schema.

Ahmet


On Thursday, July 2, 2015 11:17 AM, Søren <sd...@syntonetic.com> wrote:
Hi Solr users

I'm new to Solr and I need to be able to search in structured data in a 
case and accent insensitive manner. E.g. find "Crème brûlée", both when 
quering with "Crème brûlée" and "creme brulee".

It seems that none of the build-in text types support this, or am I wrong?
So I try to add my own inspired by another post, although it was old.

I'm running solr-5.2.1.

Curl to http://localhost:8983/solr/mycore/schema
{
"add-field-type":{
      "name":"myTxtField",
      "class":"solr.TextField",
      "positionIncrementGap":"100",
      "analyzer":{
         "charFilter": {"class":"solr.MappingCharFilterFactory", 
"mapping":"mapping-ISOLatin1Accent.txt"},
         "filter": {"class":"solr.LowerCaseFilterFactory"},
         "tokenizer": {"class":"solr.StandardTokenizerFactory"}
         }
     }
}

But it doesn't work and when I look in '[... 
]\solr-5.2.1\server\solr\mycore\conf\managed-schema'
the analyzer section is reduced to this:
   <fieldType name="myTxtField" class="solr.TextField" 
positionIncrementGap="100">
     <analyzer>
       <tokenizer class="solr.StandardTokenizerFactory"/>
     </analyzer>
   </fieldType>

  I'm I almost there or am I on a completely wrong track?

Thanks in advance
Søren