You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by vsl <oc...@gmail.com> on 2013/03/12 16:16:50 UTC
Special characters not indexed
Hi,
I am trying to index special characters and make them searchable.
User Story:
1. Index document with content: §$ %&/( )=? +*#'-<>
2. Find indexed document using search term: &
Additionaly I have several other fields that are copied to textAll Field.
The search is performed on this field.
Does anybody know how to deal with such cases?
Field definition:
<fieldType name="text_general" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true"
words="stopwords.txt" enablePositionIncrements="true" />
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.SnowballPorterFilterFactory"
language="English"/>
<filter class="solr.WordDelimiterFilterFactory"
generateWordParts="1" generateNumberParts="1" catenateWords="1"
catenateNumbers="1" catenateAll="0" splitOnCaseChange="1"
preserveOriginal="1" types="characters.txt" />
</analyzer>
<analyzer type="query">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true"
words="stopwords.txt" enablePositionIncrements="true" />
<filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt"
ignoreCase="true" expand="true"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.SnowballPorterFilterFactory"
language="English"/>
<filter class="solr.WordDelimiterFilterFactory"
generateWordParts="1" generateNumberParts="1" catenateWords="1"
catenateNumbers="1" catenateAll="0" splitOnCaseChange="1"
preserveOriginal="1" types="characters.txt" />
</analyzer>
</fieldType>
where: characters.txt
§ => ALPHA
$ => ALPHA
% => ALPHA
& => ALPHA
/ => ALPHA
( => ALPHA
) => ALPHA
= => ALPHA
? => ALPHA
+ => ALPHA
* => ALPHA
# => ALPHA
' => ALPHA
- => ALPHA
< => ALPHA
> => ALPHA
--
View this message in context: http://lucene.472066.n3.nabble.com/Special-characters-not-indexed-tp4046630.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Special characters not indexed
Posted by Timothy Potter <th...@gmail.com>.
Just to add to Jack's points, you can also use the term query parser to
avoid all the escaping for special characters, e.g.
fq={!term f=some_field}<crazy&term#value%>
See Erik's preso from Apache Eurocon 2012 around 25:50 -
http://vimeopro.com/user11514798/apache-lucene-eurocon-2012/video/55822628
On Tue, Mar 12, 2013 at 12:33 PM, Jack Krupansky <ja...@basetechnology.com>wrote:
> Use the white space tokenizer and be sure to escape a lot of them in
> queries since a number of them have meaning to the query parser. Or,
> enclose query terms in quotes.
>
> -- Jack Krupansky
>
> -----Original Message----- From: vsl
> Sent: Tuesday, March 12, 2013 11:16 AM
> To: solr-user@lucene.apache.org
> Subject: Special characters not indexed
>
>
> Hi,
> I am trying to index special characters and make them searchable.
>
> User Story:
> 1. Index document with content: §$ %&/( )=? +*#'-<>
> 2. Find indexed document using search term: &
>
> Additionaly I have several other fields that are copied to textAll Field.
> The search is performed on this field.
>
> Does anybody know how to deal with such cases?
>
> Field definition:
>
> <fieldType name="text_general" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.**StandardTokenizerFactory"/>
> <filter class="solr.StopFilterFactory" ignoreCase="true"
> words="stopwords.txt" enablePositionIncrements="**true" />
>
> <filter class="solr.**LowerCaseFilterFactory"/>
> <filter class="solr.**SnowballPorterFilterFactory"
> language="English"/>
> <filter class="solr.**WordDelimiterFilterFactory"
> generateWordParts="1" generateNumberParts="1" catenateWords="1"
> catenateNumbers="1" catenateAll="0" splitOnCaseChange="1"
> preserveOriginal="1" types="characters.txt" />
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.**StandardTokenizerFactory"/>
> <filter class="solr.StopFilterFactory" ignoreCase="true"
> words="stopwords.txt" enablePositionIncrements="**true" />
> <filter class="solr.**SynonymFilterFactory" synonyms="synonyms.txt"
> ignoreCase="true" expand="true"/>
> <filter class="solr.**LowerCaseFilterFactory"/>
> <filter class="solr.**SnowballPorterFilterFactory"
> language="English"/>
> <filter class="solr.**WordDelimiterFilterFactory"
> generateWordParts="1" generateNumberParts="1" catenateWords="1"
> catenateNumbers="1" catenateAll="0" splitOnCaseChange="1"
> preserveOriginal="1" types="characters.txt" />
> </analyzer>
> </fieldType>
>
> where: characters.txt
>
> § => ALPHA
> $ => ALPHA
> % => ALPHA
> & => ALPHA
> / => ALPHA
> ( => ALPHA
> ) => ALPHA
> = => ALPHA
> ? => ALPHA
> + => ALPHA
> * => ALPHA
> # => ALPHA
> ' => ALPHA
> - => ALPHA
> < => ALPHA
>
>> => ALPHA
>>
>
>
>
>
> --
> View this message in context: http://lucene.472066.n3.**
> nabble.com/Special-characters-**not-indexed-tp4046630.html<http://lucene.472066.n3.nabble.com/Special-characters-not-indexed-tp4046630.html>
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: Special characters not indexed
Posted by Jack Krupansky <ja...@basetechnology.com>.
If your goal is simply to determine if a character occurs within a term, you
would have to use a wildcard such as *&*.
If your goal is to make special characters be complete terms, you may have
to develop a custom tokenizer that emits multiple tokens at the same
position. Or, maybe you want to treat these complex terms as completely
separate terms. You need to be clear about what you expect.
You'll have to tell us what your goal really is.
If one of these special characters occurs embedded in a larger term, how
many terms do you wish to generate, and how do you wish them to behave in
terms of things like phrase query?
You need to be a lot more clear about your use case.
-- Jack Krupansky
-----Original Message-----
From: vsl
Sent: Wednesday, March 13, 2013 6:11 AM
To: solr-user@lucene.apache.org
Subject: Re: Special characters not indexed
After changing to white space tokenizer there are still no results for given
search term "&". Only when the whole word ("§$ %&/( )=? +*#'-<>") was given
as a search term, this document was shown in results.
--
View this message in context:
http://lucene.472066.n3.nabble.com/Special-characters-not-indexed-tp4046630p4046929.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Special characters not indexed
Posted by vsl <oc...@gmail.com>.
After changing to white space tokenizer there are still no results for given
search term "&". Only when the whole word ("§$ %&/( )=? +*#'-<>") was given
as a search term, this document was shown in results.
--
View this message in context: http://lucene.472066.n3.nabble.com/Special-characters-not-indexed-tp4046630p4046929.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Special characters not indexed
Posted by Jack Krupansky <ja...@basetechnology.com>.
Use the white space tokenizer and be sure to escape a lot of them in queries
since a number of them have meaning to the query parser. Or, enclose query
terms in quotes.
-- Jack Krupansky
-----Original Message-----
From: vsl
Sent: Tuesday, March 12, 2013 11:16 AM
To: solr-user@lucene.apache.org
Subject: Special characters not indexed
Hi,
I am trying to index special characters and make them searchable.
User Story:
1. Index document with content: §$ %&/( )=? +*#'-<>
2. Find indexed document using search term: &
Additionaly I have several other fields that are copied to textAll Field.
The search is performed on this field.
Does anybody know how to deal with such cases?
Field definition:
<fieldType name="text_general" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true"
words="stopwords.txt" enablePositionIncrements="true" />
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.SnowballPorterFilterFactory"
language="English"/>
<filter class="solr.WordDelimiterFilterFactory"
generateWordParts="1" generateNumberParts="1" catenateWords="1"
catenateNumbers="1" catenateAll="0" splitOnCaseChange="1"
preserveOriginal="1" types="characters.txt" />
</analyzer>
<analyzer type="query">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true"
words="stopwords.txt" enablePositionIncrements="true" />
<filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt"
ignoreCase="true" expand="true"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.SnowballPorterFilterFactory"
language="English"/>
<filter class="solr.WordDelimiterFilterFactory"
generateWordParts="1" generateNumberParts="1" catenateWords="1"
catenateNumbers="1" catenateAll="0" splitOnCaseChange="1"
preserveOriginal="1" types="characters.txt" />
</analyzer>
</fieldType>
where: characters.txt
§ => ALPHA
$ => ALPHA
% => ALPHA
& => ALPHA
/ => ALPHA
( => ALPHA
) => ALPHA
= => ALPHA
? => ALPHA
+ => ALPHA
* => ALPHA
# => ALPHA
' => ALPHA
- => ALPHA
< => ALPHA
> => ALPHA
--
View this message in context:
http://lucene.472066.n3.nabble.com/Special-characters-not-indexed-tp4046630.html
Sent from the Solr - User mailing list archive at Nabble.com.