You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Søren <sd...@syntonetic.com> on 2015/07/02 10:17:59 UTC
accent insensitive field-type
Hi Solr users
I'm new to Solr and I need to be able to search in structured data in a
case and accent insensitive manner. E.g. find "Crème brûlée", both when
quering with "Crème brûlée" and "creme brulee".
It seems that none of the build-in text types support this, or am I wrong?
So I try to add my own inspired by another post, although it was old.
I'm running solr-5.2.1.
Curl to http://localhost:8983/solr/mycore/schema
{
"add-field-type":{
"name":"myTxtField",
"class":"solr.TextField",
"positionIncrementGap":"100",
"analyzer":{
"charFilter": {"class":"solr.MappingCharFilterFactory",
"mapping":"mapping-ISOLatin1Accent.txt"},
"filter": {"class":"solr.LowerCaseFilterFactory"},
"tokenizer": {"class":"solr.StandardTokenizerFactory"}
}
}
}
But it doesn't work and when I look in '[...
]\solr-5.2.1\server\solr\mycore\conf\managed-schema'
the analyzer section is reduced to this:
<fieldType name="myTxtField" class="solr.TextField"
positionIncrementGap="100">
<analyzer>
<tokenizer class="solr.StandardTokenizerFactory"/>
</analyzer>
</fieldType>
I'm I almost there or am I on a completely wrong track?
Thanks in advance
Søren
Re: accent insensitive field-type
Posted by Erick Erickson <er...@gmail.com>.
bq: q=brulee didn't find anything. It goes into to the raw index I guess
The first thing to do in this case is add &debug=query to the URL and look
at the parsed query near the end of the return packet. In this case it would
look something like
q=default_search_field:brulee
where deault_search_field is the "df" specification in the request
handler in solrconfig.xml.
Best,
Erick
On Fri, Jul 3, 2015 at 1:27 AM, Søren <sd...@syntonetic.com> wrote:
> Thanks Steve! Everything works now.
> A little modification:
>
> "analyzer":{
> "charFilters": [ {"class":"solr.MappingCharFilterFactory",
> "mapping":"mapping-ISOLatin1Accent.txt"} ],
> "tokenizer": {"class":"solr.StandardTokenizerFactory"},
> "filters": [{"class":"solr.LowerCaseFilterFactory"}]
> }
>
> Thankfully, when key is a plural word, the value is an array.
>
> It was still teasing me when I tested with various queries. But specifying
> field solved that for me too.
>
> ...q=brulee didn't find anything. It goes into to the raw index I
> guess
>
> ...q=desert:brulee did find "Crème brûlée"!
>
> cheers
> Søren
>
>
>
> On 02-07-2015 17:31, Steve Rowe wrote:
>>
>> Hi Søren,
>>
>> “charFilter” should be “charFilters”, and “filter” should be “filters”;
>> and both their values should be arrays - try this:
>>
>> {
>> "add-field-type”: {
>> "name":"myTxtField",
>> "class":"solr.TextField",
>> "positionIncrementGap":"100",
>> "analyzer”: {
>> "charFilters": [ {"class":"solr.MappingCharFilterFactory",
>> "mapping":"mapping-ISOLatin1Accent.txt”} ],
>> "tokenizer": [ {"class":"solr.StandardTokenizerFactory”} ],
>> "filters": {"class":"solr.LowerCaseFilterFactory"}
>> }
>> }
>> }
>>
>> There should be better error messages for misspellings here. I’ll file a
>> JIRA issue.
>>
>> (I also moved “filters” after “tokenizer” since that’s the order in which
>> they’re executed in an analysis pipeline, but Solr will interpret the
>> out-of-order version correctly.)
>>
>> FYI, if you want to *correct* a field type, rather than create a new one,
>> you should use the “replace-field-type” command instead of the
>> “add-field-type” command. You’ll get an error if you attempt to add a field
>> type that already exists in the schema.
>>
>> Steve
>>
>>> On Jul 2, 2015, at 1:17 AM, Søren <sd...@syntonetic.com> wrote:
>>>
>>> Hi Solr users
>>>
>>> I'm new to Solr and I need to be able to search in structured data in a
>>> case and accent insensitive manner. E.g. find "Crème brûlée", both when
>>> quering with "Crème brûlée" and "creme brulee".
>>>
>>> It seems that none of the build-in text types support this, or am I
>>> wrong?
>>> So I try to add my own inspired by another post, although it was old.
>>>
>>> I'm running solr-5.2.1.
>>>
>>> Curl to http://localhost:8983/solr/mycore/schema
>>> {
>>> "add-field-type":{
>>> "name":"myTxtField",
>>> "class":"solr.TextField",
>>> "positionIncrementGap":"100",
>>> "analyzer":{
>>> "charFilter": {"class":"solr.MappingCharFilterFactory",
>>> "mapping":"mapping-ISOLatin1Accent.txt"},
>>> "filter": {"class":"solr.LowerCaseFilterFactory"},
>>> "tokenizer": {"class":"solr.StandardTokenizerFactory"}
>>> }
>>> }
>>> }
>>>
>>> But it doesn't work and when I look in '[...
>>> ]\solr-5.2.1\server\solr\mycore\conf\managed-schema'
>>> the analyzer section is reduced to this:
>>> <fieldType name="myTxtField" class="solr.TextField"
>>> positionIncrementGap="100">
>>> <analyzer>
>>> <tokenizer class="solr.StandardTokenizerFactory"/>
>>> </analyzer>
>>> </fieldType>
>>>
>>> I'm I almost there or am I on a completely wrong track?
>>>
>>> Thanks in advance
>>> Søren
>>>
>
Re: accent insensitive field-type
Posted by Steve Rowe <sa...@gmail.com>.
Hi Søren,
> On Jul 3, 2015, at 4:27 AM, Søren <sd...@syntonetic.com> wrote:
>
> Thanks Steve! Everything works now.
> A little modification:
>
> "analyzer":{
> "charFilters": [ {"class":"solr.MappingCharFilterFactory", "mapping":"mapping-ISOLatin1Accent.txt"} ],
> "tokenizer": {"class":"solr.StandardTokenizerFactory"},
> "filters": [{"class":"solr.LowerCaseFilterFactory"}]
> }
I’m glad you got it to work.
Yeah, I put square brackets in the wrong place, cool you figured it out and fixed it.
> Thankfully, when key is a plural word, the value is an array.
>
> It was still teasing me when I tested with various queries. But specifying field solved that for me too.
>
> ...q=brulee didn't find anything. It goes into to the raw index I guess
>
> ...q=desert:brulee did find "Crème brûlée”!
In your query request handler you should specify a “df” param (default field), likely under the defaults section (so that it can be overridden via per-request param) - this param will work with the dismax, edisma, or wtandard query parsers. The “qf" param, which supports a list of query fields (and field aliasing)[1], also works in the dismax and edismax query parsers.
Steve
[1] See section "Field aliasing using per-field qf overrides” on the edismax ref guide page: <https://cwiki.apache.org/confluence/display/solr/The+Extended+DisMax+Query+Parser> and the qf param description on the dismax ref guide page: <https://cwiki.apache.org/confluence/display/solr/The+DisMax+Query+Parser#TheDisMaxQueryParser-DisMaxParameters>.
Re: accent insensitive field-type
Posted by Søren <sd...@syntonetic.com>.
Thanks Steve! Everything works now.
A little modification:
"analyzer":{
"charFilters": [ {"class":"solr.MappingCharFilterFactory",
"mapping":"mapping-ISOLatin1Accent.txt"} ],
"tokenizer": {"class":"solr.StandardTokenizerFactory"},
"filters": [{"class":"solr.LowerCaseFilterFactory"}]
}
Thankfully, when key is a plural word, the value is an array.
It was still teasing me when I tested with various queries. But
specifying field solved that for me too.
...q=brulee didn't find anything. It goes into to the raw index I
guess
...q=desert:brulee did find "Crème brûlée"!
cheers
Søren
On 02-07-2015 17:31, Steve Rowe wrote:
> Hi Søren,
>
> “charFilter” should be “charFilters”, and “filter” should be “filters”; and both their values should be arrays - try this:
>
> {
> "add-field-type”: {
> "name":"myTxtField",
> "class":"solr.TextField",
> "positionIncrementGap":"100",
> "analyzer”: {
> "charFilters": [ {"class":"solr.MappingCharFilterFactory", "mapping":"mapping-ISOLatin1Accent.txt”} ],
> "tokenizer": [ {"class":"solr.StandardTokenizerFactory”} ],
> "filters": {"class":"solr.LowerCaseFilterFactory"}
> }
> }
> }
>
> There should be better error messages for misspellings here. I’ll file a JIRA issue.
>
> (I also moved “filters” after “tokenizer” since that’s the order in which they’re executed in an analysis pipeline, but Solr will interpret the out-of-order version correctly.)
>
> FYI, if you want to *correct* a field type, rather than create a new one, you should use the “replace-field-type” command instead of the “add-field-type” command. You’ll get an error if you attempt to add a field type that already exists in the schema.
>
> Steve
>
>> On Jul 2, 2015, at 1:17 AM, Søren <sd...@syntonetic.com> wrote:
>>
>> Hi Solr users
>>
>> I'm new to Solr and I need to be able to search in structured data in a case and accent insensitive manner. E.g. find "Crème brûlée", both when quering with "Crème brûlée" and "creme brulee".
>>
>> It seems that none of the build-in text types support this, or am I wrong?
>> So I try to add my own inspired by another post, although it was old.
>>
>> I'm running solr-5.2.1.
>>
>> Curl to http://localhost:8983/solr/mycore/schema
>> {
>> "add-field-type":{
>> "name":"myTxtField",
>> "class":"solr.TextField",
>> "positionIncrementGap":"100",
>> "analyzer":{
>> "charFilter": {"class":"solr.MappingCharFilterFactory", "mapping":"mapping-ISOLatin1Accent.txt"},
>> "filter": {"class":"solr.LowerCaseFilterFactory"},
>> "tokenizer": {"class":"solr.StandardTokenizerFactory"}
>> }
>> }
>> }
>>
>> But it doesn't work and when I look in '[... ]\solr-5.2.1\server\solr\mycore\conf\managed-schema'
>> the analyzer section is reduced to this:
>> <fieldType name="myTxtField" class="solr.TextField" positionIncrementGap="100">
>> <analyzer>
>> <tokenizer class="solr.StandardTokenizerFactory"/>
>> </analyzer>
>> </fieldType>
>>
>> I'm I almost there or am I on a completely wrong track?
>>
>> Thanks in advance
>> Søren
>>
Re: accent insensitive field-type
Posted by Steve Rowe <sa...@gmail.com>.
See https://issues.apache.org/jira/browse/SOLR-7749
> On Jul 2, 2015, at 8:31 AM, Steve Rowe <sa...@gmail.com> wrote:
>
> Hi Søren,
>
> “charFilter” should be “charFilters”, and “filter” should be “filters”; and both their values should be arrays - try this:
>
> {
> "add-field-type”: {
> "name":"myTxtField",
> "class":"solr.TextField",
> "positionIncrementGap":"100",
> "analyzer”: {
> "charFilters": [ {"class":"solr.MappingCharFilterFactory", "mapping":"mapping-ISOLatin1Accent.txt”} ],
> "tokenizer": [ {"class":"solr.StandardTokenizerFactory”} ],
> "filters": {"class":"solr.LowerCaseFilterFactory"}
> }
> }
> }
>
> There should be better error messages for misspellings here. I’ll file a JIRA issue.
>
> (I also moved “filters” after “tokenizer” since that’s the order in which they’re executed in an analysis pipeline, but Solr will interpret the out-of-order version correctly.)
>
> FYI, if you want to *correct* a field type, rather than create a new one, you should use the “replace-field-type” command instead of the “add-field-type” command. You’ll get an error if you attempt to add a field type that already exists in the schema.
>
> Steve
>
>> On Jul 2, 2015, at 1:17 AM, Søren <sd...@syntonetic.com> wrote:
>>
>> Hi Solr users
>>
>> I'm new to Solr and I need to be able to search in structured data in a case and accent insensitive manner. E.g. find "Crème brûlée", both when quering with "Crème brûlée" and "creme brulee".
>>
>> It seems that none of the build-in text types support this, or am I wrong?
>> So I try to add my own inspired by another post, although it was old.
>>
>> I'm running solr-5.2.1.
>>
>> Curl to http://localhost:8983/solr/mycore/schema
>> {
>> "add-field-type":{
>> "name":"myTxtField",
>> "class":"solr.TextField",
>> "positionIncrementGap":"100",
>> "analyzer":{
>> "charFilter": {"class":"solr.MappingCharFilterFactory", "mapping":"mapping-ISOLatin1Accent.txt"},
>> "filter": {"class":"solr.LowerCaseFilterFactory"},
>> "tokenizer": {"class":"solr.StandardTokenizerFactory"}
>> }
>> }
>> }
>>
>> But it doesn't work and when I look in '[... ]\solr-5.2.1\server\solr\mycore\conf\managed-schema'
>> the analyzer section is reduced to this:
>> <fieldType name="myTxtField" class="solr.TextField" positionIncrementGap="100">
>> <analyzer>
>> <tokenizer class="solr.StandardTokenizerFactory"/>
>> </analyzer>
>> </fieldType>
>>
>> I'm I almost there or am I on a completely wrong track?
>>
>> Thanks in advance
>> Søren
>>
>
Re: accent insensitive field-type
Posted by Steve Rowe <sa...@gmail.com>.
Hi Søren,
“charFilter” should be “charFilters”, and “filter” should be “filters”; and both their values should be arrays - try this:
{
"add-field-type”: {
"name":"myTxtField",
"class":"solr.TextField",
"positionIncrementGap":"100",
"analyzer”: {
"charFilters": [ {"class":"solr.MappingCharFilterFactory", "mapping":"mapping-ISOLatin1Accent.txt”} ],
"tokenizer": [ {"class":"solr.StandardTokenizerFactory”} ],
"filters": {"class":"solr.LowerCaseFilterFactory"}
}
}
}
There should be better error messages for misspellings here. I’ll file a JIRA issue.
(I also moved “filters” after “tokenizer” since that’s the order in which they’re executed in an analysis pipeline, but Solr will interpret the out-of-order version correctly.)
FYI, if you want to *correct* a field type, rather than create a new one, you should use the “replace-field-type” command instead of the “add-field-type” command. You’ll get an error if you attempt to add a field type that already exists in the schema.
Steve
> On Jul 2, 2015, at 1:17 AM, Søren <sd...@syntonetic.com> wrote:
>
> Hi Solr users
>
> I'm new to Solr and I need to be able to search in structured data in a case and accent insensitive manner. E.g. find "Crème brûlée", both when quering with "Crème brûlée" and "creme brulee".
>
> It seems that none of the build-in text types support this, or am I wrong?
> So I try to add my own inspired by another post, although it was old.
>
> I'm running solr-5.2.1.
>
> Curl to http://localhost:8983/solr/mycore/schema
> {
> "add-field-type":{
> "name":"myTxtField",
> "class":"solr.TextField",
> "positionIncrementGap":"100",
> "analyzer":{
> "charFilter": {"class":"solr.MappingCharFilterFactory", "mapping":"mapping-ISOLatin1Accent.txt"},
> "filter": {"class":"solr.LowerCaseFilterFactory"},
> "tokenizer": {"class":"solr.StandardTokenizerFactory"}
> }
> }
> }
>
> But it doesn't work and when I look in '[... ]\solr-5.2.1\server\solr\mycore\conf\managed-schema'
> the analyzer section is reduced to this:
> <fieldType name="myTxtField" class="solr.TextField" positionIncrementGap="100">
> <analyzer>
> <tokenizer class="solr.StandardTokenizerFactory"/>
> </analyzer>
> </fieldType>
>
> I'm I almost there or am I on a completely wrong track?
>
> Thanks in advance
> Søren
>
Re: accent insensitive field-type
Posted by Shawn Heisey <ap...@elyograg.org>.
On 7/2/2015 8:53 AM, Søren wrote:
> I'm trying the add the ICUFoldingFilterFactory in the analyzer.
> I suspect that my problem is that the filter class doesn't load.
> The managed-schema file is the same info as when looking in the schema
> browser in the web gui.
The ICU analysis components are considered "contrib" and therefore
require adding jars to Solr for them to work. The notes on the
AnalyzersTokenizersTokenFilters wiki page tell you where to look for
more information about those jars.
http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters#solr.ICUFoldingFilterFactory
The best place to add additional jars is ${solr.solr.home}/lib ... a
directory which usually must be created. The solr home is where
solr.xml lives, providing global config options to Solr. The default
location for the solr home in the 5.x download is server/solr, but that
can be overridden.
Thanks,
Shawn
Re: accent insensitive field-type
Posted by Søren <sd...@syntonetic.com>.
Thanks Ahmet
I'm trying the add the ICUFoldingFilterFactory in the analyzer.
I suspect that my problem is that the filter class doesn't load.
The managed-schema file is the same info as when looking in the schema
browser in the web gui.
Cheers
On 02-07-2015 10:47, Ahmet Arslan wrote:
> Hi Soren,
>
> I am not familiar with managed schema part, but there are built-in filters for this task.
>
> ASCIIFoldingFilter and ICUFoldingFilter are two examples.
>
> Also solr provides two files: mapping-FoldToASCII.txt and mapping-ISOLatin1Accent.txt to be used with
> MappingCharFilter as you did.
> You are probably hitting a problem with managed schema.
>
> Ahmet
>
>
> On Thursday, July 2, 2015 11:17 AM, Søren <sd...@syntonetic.com> wrote:
> Hi Solr users
>
> I'm new to Solr and I need to be able to search in structured data in a
> case and accent insensitive manner. E.g. find "Crème brûlée", both when
> quering with "Crème brûlée" and "creme brulee".
>
> It seems that none of the build-in text types support this, or am I wrong?
> So I try to add my own inspired by another post, although it was old.
>
> I'm running solr-5.2.1.
>
> Curl to http://localhost:8983/solr/mycore/schema
> {
> "add-field-type":{
> "name":"myTxtField",
> "class":"solr.TextField",
> "positionIncrementGap":"100",
> "analyzer":{
> "charFilter": {"class":"solr.MappingCharFilterFactory",
> "mapping":"mapping-ISOLatin1Accent.txt"},
> "filter": {"class":"solr.LowerCaseFilterFactory"},
> "tokenizer": {"class":"solr.StandardTokenizerFactory"}
> }
> }
> }
>
> But it doesn't work and when I look in '[...
> ]\solr-5.2.1\server\solr\mycore\conf\managed-schema'
> the analyzer section is reduced to this:
> <fieldType name="myTxtField" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer>
> <tokenizer class="solr.StandardTokenizerFactory"/>
> </analyzer>
> </fieldType>
>
> I'm I almost there or am I on a completely wrong track?
>
> Thanks in advance
> Søren
Re: accent insensitive field-type
Posted by Ahmet Arslan <io...@yahoo.com.INVALID>.
Hi Soren,
I am not familiar with managed schema part, but there are built-in filters for this task.
ASCIIFoldingFilter and ICUFoldingFilter are two examples.
Also solr provides two files: mapping-FoldToASCII.txt and mapping-ISOLatin1Accent.txt to be used with
MappingCharFilter as you did.
You are probably hitting a problem with managed schema.
Ahmet
On Thursday, July 2, 2015 11:17 AM, Søren <sd...@syntonetic.com> wrote:
Hi Solr users
I'm new to Solr and I need to be able to search in structured data in a
case and accent insensitive manner. E.g. find "Crème brûlée", both when
quering with "Crème brûlée" and "creme brulee".
It seems that none of the build-in text types support this, or am I wrong?
So I try to add my own inspired by another post, although it was old.
I'm running solr-5.2.1.
Curl to http://localhost:8983/solr/mycore/schema
{
"add-field-type":{
"name":"myTxtField",
"class":"solr.TextField",
"positionIncrementGap":"100",
"analyzer":{
"charFilter": {"class":"solr.MappingCharFilterFactory",
"mapping":"mapping-ISOLatin1Accent.txt"},
"filter": {"class":"solr.LowerCaseFilterFactory"},
"tokenizer": {"class":"solr.StandardTokenizerFactory"}
}
}
}
But it doesn't work and when I look in '[...
]\solr-5.2.1\server\solr\mycore\conf\managed-schema'
the analyzer section is reduced to this:
<fieldType name="myTxtField" class="solr.TextField"
positionIncrementGap="100">
<analyzer>
<tokenizer class="solr.StandardTokenizerFactory"/>
</analyzer>
</fieldType>
I'm I almost there or am I on a completely wrong track?
Thanks in advance
Søren